Video Search Research Articles

Interactive Search vs. Automatic Search

This article conducts user evaluation to study the performance difference between interactive and automatic search. Particularly, the study aims to provide empirical insights of how the performance landscape of video search changes, with tens of thousands of concept detectors freely available to exploit for query formulation. We compare three types of search modes: free-to-play (i.e., search from scratch), non-free-to-play (i.e., search by inspecting results provided by automatic search), and automatic search including concept-free and concept-based retrieval paradigms. The study involves a total of 40 participants; each performs interactive search over 15 queries of various difficulty levels using two search modes on the IACC.3 dataset provided by TRECVid organizers. The study suggests that the performance of automatic search is still far behind interactive search. Furthermore, providing users with the result of automatic search for exploration does not show obvious advantage over asking users to search from scratch. The study also analyzes user behavior to reveal insights of how users compose queries, browse results, and discover new query terms for search, which can serve as guideline for future research of both interactive and automatic search.

The VISIONE Video Search System: Exploiting Off-the-Shelf Text Search Engines for Large-Scale Video Retrieval.

This paper describes in detail VISIONE, a video search system that allows users to search for videos using textual keywords, the occurrence of objects and their spatial relationships, the occurrence of colors and their spatial relationships, and image similarity. These modalities can be combined together to express complex queries and meet users’ needs. The peculiarity of our approach is that we encode all information extracted from the keyframes, such as visual deep features, tags, color and object locations, using a convenient textual encoding that is indexed in a single text retrieval engine. This offers great flexibility when results corresponding to various parts of the query (visual, text and locations) need to be merged. In addition, we report an extensive analysis of the retrieval performance of the system, using the query logs generated during the Video Browser Showdown (VBS) 2019 competition. This allowed us to fine-tune the system by choosing the optimal parameters and strategies from those we tested.

Zero-Shot Video Event Detection With High-Order Semantic Concept Discovery and Matching

Multimedia event detection aims to precisely retrieve videos that contain complex semantic events from a large pool. This work addresses this task under a zero-shot setting, where only brief event-specific textural information (such as event names, a few descriptive sentences, etc.) is known yet none positive video example is provided. Mainstream approaches to tackling this task are middle-level semantic concept-based, where meticulously-crafted concept banks (e.g., LSCOM) are adopted. We argue that these concept banks are still inadequate facing video semantic complexity. Existing semantic concepts are essentially first-order, mainly designed for atomic objects, scenes or human actions, etc. This work advocates the utilization of high-order concepts (such as subject-predicate-object triplets or adjective-object). The main contributions are two-fold. First, we harvest a comprehensive albeit compact high-order concept library through distilling information from three large public datasets (MS-COCO, Visual Genome, and Kinetics-600), mainly related to visual relations and human-object interactions. Secondly, zero-shot events are often only briefly and partially described via textual input. The resultant semantic ambiguity makes the pursuit of the most indicative high-order concepts challenging. We thus design a novel query-expanding scheme that enriches ambiguous event-specific keywords by searching over either large common knowledge bases ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , WikiHow) or top-ranked webpages retrieved from modern search engines. This way sets up a more faithful connection between zero-shot events and high-order concepts. To our best knowledge, this is the first work that strives for concept-based video search beyond first-order concepts. Extensive experiments have been conducted on several large video benchmarks (TRECVID 2013, TRECVID 2014, and ActivityNet-1.3). The evaluations clearly demonstrate the superiority of our constructed high-order concept library and its complementariness to existing concepts.

Design a model of Image Restoration using AI in Digital Image Processing

Image restoration is the process of obtaining a distorted/noise image and giving an approximate clear image of the original image. False focus, motion blur and noise are forms of distortion. Image restoration can be done by reversing the process called Point Extension Function (PSF). In this process, the blurred image is generated by point source imaging and can be used to restore the image lost due to the blur process. Like to form. Modern artificial intelligence (AI) applied to image processing includes facial recognition, object recognition and detection, video, image action, and visual search. It helps to develop smart applications in digital image processing.

On the User-Centric Comparative Remote Evaluation of Interactive Video Search Systems

In the research of video retrieval systems, comparative assessments during dedicated retrieval competitions provide priceless insights into the performance of individual systems. The scope and depth of such evaluations are unfortunately hard to improve, due to the limitations by the set-up costs, logistics, and organization complexity of large events. We show that this easily impairs the statistical significance of the collected results, and the reproducibility of the competition outcomes. In this article, we present a methodology for remote comparative evaluations of content-based video retrieval systems and demonstrate that such evaluations scale-up to sizes that reliably produce statistically robust results, and propose additional measures that increase the replicability of the experiment. The proposed remote evaluation methodology forms a major contribution toward open science in interactive retrieval benchmarks. At the same time, the detailed evaluation reports form an interesting source of new observations about many subtle, previously inaccessible aspects of video retrieval.

Nine typical injury patterns in German professional male football (soccer): a systematic visual video analysis of 345 match injuries

AimWe aimed to systematically analyse the videos of acute injuries in professional men’s football and describe typical injury patterns.MethodsInjuries were registered with the German statutory accident insurance for professional athletes...

Multi-level Alignment Network for Domain Adaptive Cross-modal Retrieval

Cross-modal retrieval is an important but challenging research task in the multimedia community. Most existing works of this task are supervised, which typically train models on a large number of aligned image-text/video-text pairs, making an assumption that training and testing data are drawn from the same distribution. If this assumption does not hold, traditional cross-modal retrieval methods may experience a performance drop at the evaluation. In this paper, we introduce a new task named as domain adaptive cross-modal retrieval, where training (source) data and testing (target) data are from different domains. The task is challenging, as there are not only the semantic gap and modality gap between visual and textual items, but also domain gap between source and target domains. Therefore, we propose a Multi-level Alignment Network (MAN) that has two mapping modules to project visual and textual modalities in a common space respectively, and three alignments are used to learn more discriminative features in the space. A semantic alignment is used to reduce the semantic gap, a cross-modality alignment and a cross-domain alignment are employed to alleviate the modality gap and domain gap. Extensive experiments in the context of domain-adaptive image-text retrieval and video-text retrieval demonstrate that our proposed model, MAN, consistently outperforms multiple baselines, showing a superior generalization ability for target data. Moreover, MAN establishes a new state-of-the-art for the large-scale text-to-video retrieval on TRECVID 2017, 2018 Ad-hoc Video Search benchmark.

Cruising the Hyper-Real Highway: Edgar Wright’s Baby Driver

Research Article| April 01 2021 Cruising the Hyper-Real Highway: Edgar Wright’s Baby Driver Derek Dubois Derek Dubois Derek Dubois is an award-winning writer and filmmaker and an adjunct professor of film studies at Rhode Island College and Clark University. His work has appeared in the peer-reviewed journals Film International, Short Film Studies, Supernatural Studies, and others. Most recently, his literary nonfiction piece “Little Swimmers” was selected as a notable work by the 2019 Best American Essays series. Search for other works by this author on: This Site Google Journal of Film and Video (2021) 73 (1): 48–58. https://doi.org/10.5406/jfilmvideo.73.1.0048 Cite Icon Cite Share Icon Share Twitter Permissions Search Site Citation Derek Dubois; Cruising the Hyper-Real Highway: Edgar Wright’s Baby Driver. Journal of Film and Video 1 January 2021; 73 (1): 48–58. doi: https://doi.org/10.5406/jfilmvideo.73.1.0048 Download citation file: Zotero Reference Manager EasyBib Bookends Mendeley Papers EndNote RefWorks BibTex toolbar search Search Dropdown Menu toolbar search search input Search input auto suggest filter your search All Scholarly Publishing CollectiveUniversity of Illinois PressJournal of Film and Video Search Advanced Search The text of this article is only available as a PDF. Copyright 2021 by the Board of Trustees of the University of Illinois2021 Article PDF first page preview Close Modal You do not currently have access to this content.

“I’m Taken … by Myself”: Romantic Crisis in the Self-Centered Indie Rom-Com

Research Article| April 01 2021 “I’m Taken … by Myself”: Romantic Crisis in the Self-Centered Indie Rom-Com Beatriz Oria Beatriz Oria Beatriz Oria is associate professor of film studies at the University of Zaragoza, where she teaches film analysis. Her current research focuses on contemporary romantic comedy and US independent cinema. Search for other works by this author on: This Site Google Journal of Film and Video (2021) 73 (1): 3–17. https://doi.org/10.5406/jfilmvideo.73.1.0003 Cite Icon Cite Share Icon Share Twitter Permissions Search Site Citation Beatriz Oria; “I’m Taken … by Myself”: Romantic Crisis in the Self-Centered Indie Rom-Com. Journal of Film and Video 1 January 2021; 73 (1): 3–17. doi: https://doi.org/10.5406/jfilmvideo.73.1.0003 Download citation file: Zotero Reference Manager EasyBib Bookends Mendeley Papers EndNote RefWorks BibTex toolbar search Search Dropdown Menu toolbar search search input Search input auto suggest filter your search All Scholarly Publishing CollectiveUniversity of Illinois PressJournal of Film and Video Search Advanced Search The text of this article is only available as a PDF. Copyright 2021 by the Board of Trustees of the University of Illinois2021 Article PDF first page preview Close Modal You do not currently have access to this content.

Deep Learning Networks-Based Action Videos Classification and Search

This work presents the deep learning networks-based method using fine-tuning for classification and search of a diversity of action videos. First, a 3D convolutional neural networks (3D CNN) model which performs pre-training operation and fine-tuning strategy is employed to extract the spatiotemporal features of videos. It is first pre-trained on UCF-101 datasets to train model with initial parameters. Then, a small new dataset is employed to fine-tune the initial model for the training of the new model. Once features are extracted by the final CNNs model, distance measure can be adopted to calculate the similarities between the query video and the test dataset for the video search. The searched video is returned and ranked according to the priority when it has higher similarity with the query video. The comparison results in the experiment shows that the search method using fine-tuning obtains better performance than the method without using fine-tuning. Second, the classification results based on the 3D CNN model using fine-tuning are also presented for the consideration of a query by keyword. Accuracy result obtained using the model with the help of fine-tuning is approximately 2.8% higher than that without using fine-tuning.

Indexed Dataset from YouTube for a Content-Based Video Search Engine

Numerous researches on content-based video indexing and retrieval besides video search engines are tied to a large-scaled video dataset. Unfortunately, reduction in open-sourced datasets resulted in complications for novel approaches exploration. Although, video datasets that index video files located on public video streaming services have other purposes, such as annotation, learning, classification, and other computer vision areas, with little interest in indexing public video links for purpose of searching and retrieval. This paper introduces a novel large-scaled dataset based on YouTube video links to evaluate the proposed content-based video search engine, gathered 1088 videos, that represent more than 65 hours of video, 11,000 video shots, and 66,000 unmarked and marked keyframes, 80 different object names used for marking. Moreover, a state-of-the-art features vector, and combinational-based matching, beneficial to the accuracy, speed, and precision of the video retrieval process. Any video record in the dataset is represented by three features: temporal combination vector, object combination vector with shot annotations, and 6 keyframes, sideways with other metadata. Video classification for the dataset was also imposed to expand the efficiency of retrieval of video-based queries. A two-phased approach has been used based on object and event classification, storing video records in aggregations related to feature vectors extracted. While object aggregation stores video records with the maximal occurrence of extracted object/concept from all shots, event aggregation classify based on groups according to the number of shots per video. This study indexed 58 out of 80 different object/concept categories, each has 9 shot number groups.

Olympic Games Event Recognition via Transfer Learning with Photobombing Guided Data Augmentation.

Automatic event recognition in sports photos is both an interesting and valuable research topic in the field of computer vision and deep learning. With the rapid increase and the explosive spread of data, which is being captured momentarily, the need for fast and precise access to the right information has become a challenging task with considerable importance for multiple practical applications, i.e., sports image and video search, sport data analysis, healthcare monitoring applications, monitoring and surveillance systems for indoor and outdoor activities, and video captioning. In this paper, we evaluate different deep learning models in recognizing and interpreting the sport events in the Olympic Games. To this end, we collect a dataset dubbed Olympic Games Event Image Dataset (OGED) including 10 different sport events scheduled for the Olympic Games Tokyo 2020. Then, the transfer learning is applied on three popular deep convolutional neural network architectures, namely, AlexNet, VGG-16 and ResNet-50 along with various data augmentation methods. Extensive experiments show that ResNet-50 with the proposed photobombing guided data augmentation achieves 90% in terms of accuracy.

Evaluation of educational value of YouTube videos addressing robotic pyeloplasty in children

Ureteropelvic junction obstruction (UPJO) is the most frequent obstructive uropathy of the upper urinary tract in children. Video-sharing platforms have become a significant source of visual information for health care providers. Among these platforms, YouTube® (www.youtube.com), contains a high number of videos free of charge and represents one of the most important and known websites of video-sharing. YouTube® is a widely used open-access video sharing website that allows us to watch an unlimited number of video content, and to upload an infinitive number of videos. This study aims to evaluate the educational quality of videos related to robotic pyeloplasty in pediatric age because an increasing number of videos addressing these procedures is now available on YouTube®. We performed a search on YouTube® by using the following keyword: "robotic pyeloplasty in children" on July 9, 2020. The first 50 videos were analyzed. The videos were classified according to the source in 1) academic (author/s was/were affiliated with a university), 2) physician (author/s who was/were not affiliated with a university), 3) patient, 4) commercial. All the videos were evaluated also according to the content in 1) surgical technique, 2) information about the surgery and disease 3) patient personal experience 4) advertisement. Duplicated videos and videos not in English were excluded. The search for videos was done based on the website's default settings in order of the proposed relevance. The reliability was evaluated using DISCREN and JAMA scores. The Global Quality Score (GWS) was used to assess the educational value. Time since upload, run time, like, dislike and number of views were recorded. The first 50 videos were analyzed. Seven videos (14%) did not meet our criteria and were excluded (three videos were duplicated while 4 out of seven were not in English). The mean DISCERN was 32.47±12.24 (range 15-78). The mean JAMA Score was 2.1±0.9 (range 0-4). Mean GQS was 2,12±0.9. DISCERN and JAMAS and GQS scores of academic/physician sourced videos were significantly higher than the patient sourced videos (p=0.037, p=0.023, p=0.017 respectively). Regarding content, the surgical technique had significantly higher DISCERN, JAMAS and GQS scores than videos based on patient experience (p=0.012, p: 0.021, p=0.023 respectively). Videos uploaded by Physicians and Academic Institutions show higher DISCERN and JAMAS and GQS compared to other sourced videos and should be considered more suitable for teaching respect to those originating from patients or non-physicians.

A content based image information retrieval and video thumbnail extraction framework using SOM

Searching an image or a video in a huge volume of graphical data is a tedious time-consuming process. If this search is performed using the conventional element matching technique, the complexity of the search will render the system useless. To overcome this problem, the current paper proposes a Content-Based Image Retrieval (CBIR) and a Content-Based Video Retrieval (CBVR) technique using clustering algorithms based on neural networks. Neural networks have proved to be quite powerful for dimensionality reduction due to their parallel computations. Retrieval of images in a large database on the basis of the content of the query image has been proved fast and efficient through practical results. Two images of the same object, but taken from different camera angles or have rotational and scaling transforms is also matched effectively. In medical domain, CBIR has proved to be a boon to the doctors. The tumor, cancer etc can be easily deducted comparing the images with normal to the images with diseases. Java and Weka have been used for implementation. The thumbnails extracted from the video facilitates the video search in a large videos database. The unsupervised nature of Self Organizing Maps (SOM) has made the software all the more robust.

Assessment of Quality and Reliability of Electroconvulsive Therapy Videos Available on YouTube

Objective: ECT is still an important treatment modality in contemporary psychiatry. Despite the evidence regarding its efficacy and safety, patients are hesitant about ECT, which is sometimes stigmatized due to disinformation. Today, YouTube, an online media platform, is mostly used as a medical information source. This study investigated the quality and reliability of electroconvulsive therapy videos available on YouTube. Methods: A video search on YouTube (http://www.youtube.com) was conducted on 20 October 2020, using the keywords electroconvulsive therapy schizophrenia, electroconvulsive therapy depression, and electroconvulsive therapy psychiatric disorders in this descriptive study. After applying the exclusion criteria, a total of 79 videos were included in this study. The Global Quality Scale (GQS) was used to assess educational quality and the modified DISCERN tool was used to assess reliability. Video parameters (the numbers of views, likes, dislikes, and comments for per day) were compared among the low/medium/high quality groups. Results: 45.2% (n: 42) of the videos were of low quality, 25.8% (n: 24) were of medium quality and 14% (n: 13) were of high quality. High-quality videos were 38.5% from psychiatrists, 23.1% from academic resources, 23.1% from health-related websites. Modified DISCERN scores were higher in the high quality video group (p

FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos

Recording and sharing of educational or lecture videos has increased in recent years. Within these recordings, we find a large number of math-oriented lectures and tutorials which attract students of all levels. Many of the topics covered by these recordings are better explained using handwritten content on whiteboards or chalkboards. Hence, we find large numbers of lecture videos that feature the instructor writing on a surface. In this work, we propose a novel method for extraction and summarization of the handwritten content found in such videos. Our method is based on a fully convolutional network, FCN-LectureNet, which can extract the handwritten content from the video as binary images. These are further analyzed to identify the unique and stable units of content to produce a spatial-temporal index of handwritten content. A signal which approximates content deletion events is then built using information from the spatial-temporal index. The peaks of this signal are used to create temporal segments of the lecture based on the notion that sub-topics change when large portions of content are deleted. Finally, we use these segments to create an extractive summary of the handwritten content based on key-frames. This will facilitate content-based search and retrieval of these lecture videos. In this work, we also extend the AccessMath dataset to create a novel dataset for benchmarking of lecture video summarization called LectureMath. Our experiments on both datasets show that our novel method can outperform existing methods especially on the larger and more challenging dataset. Our code and data are publicly available.

Reader service experience system of intelligent library based on artificial intelligence and machine learning

With the development of social economy and the improvement of science and technology, digital video on the Internet is increasing rapidly, and it has become a new force to promote the development of the times. Most of these videos are stored in the memory, which poses a great challenge to the research and development of the system. The reader service system is an important part of library service. The library uses it to collect information resources, not just for service and work. The document combines the video of library service, the analysis of video recovery and video software requirements of digital library, puts forward the design goal and conception of video search, and puts forward a foundation. From the video data of digital library, video retrieval experiments are gradually carried out. These experimental results show that the number of enhanced dynamic clustering algorithm increases to ensure the complexity of the image.

Place Classification Algorithm Based on Semantic Segmented Objects

Scene or place classification is one of the important problems in image and video search and recommendation systems. Humans can understand the scene they are located, but it is difficult for machines to do it. Considering a scene image which has several objects, humans recognize the scene based on these objects, especially background objects. According to this observation, we propose an efficient scene classification algorithm for three different classes by detecting objects in the scene. We use pre-trained semantic segmentation model to extract objects from an image. After that, we construct a weight matrix to determine a scene class better. Finally, we classify an image into one of three scene classes (i.e., indoor, nature, city) by using the designed weighting matrix. The performance of our scheme outperforms several classification methods using convolutional neural networks (CNNs), such as VGG, Inception, ResNet, ResNeXt, Wide-ResNet, DenseNet, and MnasNet. The proposed model achieves 90.8% of verification accuracy and improves over 2.8% of the accuracy when comparing to the existing CNN-based methods.

SEA: Sentence Encoder Assembly for Video Retrieval by Textual Queries

Retrieving unlabeled videos by textual queries, known as Ad-hoc Video Search (AVS), is a core theme in multimedia data management and retrieval. The success of AVS counts on cross-modal representation learning that encodes both query sentences and videos into common spaces for semantic similarity computation. Inspired by the initial success of previously few works in combining multiple sentence encoders, this paper takes a step forward by developing a new and general method for effectively exploiting diverse sentence encoders. The novelty of the proposed method, which we term <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Sentence Encoder Assembly</i> (SEA), is two-fold. First, different from prior art that uses only a single common space, SEA supports text-video matching in multiple encoder-specific common spaces. Such a property prevents the matching from being dominated by a specific encoder that produces an encoding vector much longer than other encoders. Second, in order to explore complementarities among the individual common spaces, we propose multi-space multi-loss learning. As extensive experiments on four benchmarks (MSR-VTT, TRECVID AVS 2016-2019, TGIF and MSVD) show, SEA surpasses the state-of-the-art. In addition, SEA is extremely ease to implement. All this makes SEA an appealing solution for AVS and promising for continuously advancing the task by harvesting new sentence encoders.

Oedipus the King as a Paradigm for Family Trauma Cinema

Research Article| October 01 2020 Oedipus the King as a Paradigm for Family Trauma Cinema Gail Finney Gail Finney Gail Finney is a professor of comparative literature and German at the University of California, Davis. She taught previously at Harvard University. She has published three monographs and edited or coedited five volumes in the fields of nineteenth- and twentieth-century German/European literature and trauma studies. Search for other works by this author on: This Site Google Journal of Film and Video (2020) 72 (3-4): 64–72. https://doi.org/10.5406/jfilmvideo.72.3-4.0064 Cite Icon Cite Share Icon Share Twitter Permissions Search Site Citation Gail Finney; Oedipus the King as a Paradigm for Family Trauma Cinema. Journal of Film and Video 1 January 2020; 72 (3-4): 64–72. doi: https://doi.org/10.5406/jfilmvideo.72.3-4.0064 Download citation file: Zotero Reference Manager EasyBib Bookends Mendeley Papers EndNote RefWorks BibTex toolbar search Search Dropdown Menu toolbar search search input Search input auto suggest filter your search All Scholarly Publishing CollectiveUniversity of Illinois PressJournal of Film and Video Search Advanced Search The text of this article is only available as a PDF. Copyright 2020 by the Board of Trustees of the University of Illinois2020 Article PDF first page preview Close Modal You do not currently have access to this content.