Video Search Engine Research Articles

Algorithm for Extraction of Keyframes of Images in Video Retrieval Systems

As a part of this work, there was a study of image processing algorithms used in video search systems.With the development of search engines and an increase in the types of queries possible for searching, the need for indexing an increasing amount of diverse information is growing. New data in the form of images and videos require new processing techniques to extract key content descriptions. In video search engines, according to this description, users can find the video files most relevant to the search query. The search query, in turn, can be of various types: text, search by image, search by video file to find a similar one, etc. Therefore, it is necessary to accurately describe the objects in the video in order to assign appropriate labels to the video file in the search engine database.In this article, we focused on the algorithm for extracting key frames of faces from a video sequence, since one of the important objects in the video are people themselves. This algorithm allows you to perform the initial processing of the file and save the identified frames with faces in order to later process this data with the help of the face recognition algorithm and assign the appropriate labels. An alternative application for this algorithm is the current processing of video files to form datasets of faces for the development and training of new computer vision models. The main criteria for such an algorithm were: the accuracy of face detection, the ability to distinguish keyframes of all people from each other, comprehensive evaluation of candidate frames and sorting by the relevance of the entire set for each face.After an analysis of existing solutions for specific stages of the algorithm, the article proposes a sequence of steps for the algorithm for extracting key frames of faces from a video file. An important step is to assess the quality of all candidates and sort them by quality. For this, the work defines various metrics for assessing the quality of the frame, which affect the overall assessment and, accordingly, the sorting order. The article also describes the basic version of the interface for using the proposed algorithm.

Development of an Effective Bootleg Videos Retrieval System as a Part of Content-Based Video Search Engine

Many research studies in content-based video search engines are concerned with content-based video queries retrieval where a query by example is sent to retrieve a list of visually similar videos. However, minor research is concerned with indexing and searching public video streaming services such as YouTube, where there is a dilemma for misusing copyrighted video materials and detecting bootleg manipulated videos before being uploaded. In this paper, a novel and effective technique for a content-based video search engine with effective detection of bootleg videos is evaluated on a large-scale video index dataset of 1088 video records. A novel feature vector is introduced using video shots temporal and key-object/concept features applying combinational-based matching algorithms, using various similarity metrics for evaluation. The retrieval system was evaluated using more than 200 non-semantic-based video queries evaluating both normal and bootleg videos, with retrieval precision for normal videos of 97.9% and retrieval recall of 100% combined by the F1 measure to be 98.3%. Bootleg videos retrieval precision scored 99.2% and retrieval recall was of 96.7% combined by the F1 measure to be 97.9%. This allows making a conclusion that this technique can help in enhancing both traditional text-based search engines and commonly used bootleg detection techniques.

Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown.

The Video Browser Showdown addresses difficult video search challenges through an annual interactive evaluation campaign attracting research teams focusing on interactive video retrieval. The campaign aims to provide insights into the performance of participating interactive video retrieval systems, tested by selected search tasks on large video collections. For the first time in its ten year history, the Video Browser Showdown 2021 was organized in a fully remote setting and hosted a record number of sixteen scoring systems. In this paper, we describe the competition setting, tasks and results and give an overview of state-of-the-art methods used by the competing systems. By looking at query result logs provided by ten systems, we analyze differences in retrieval model performances and browsing times before a correct submission. Through advances in data gathering methodology and tools, we provide a comprehensive analysis of ad-hoc video search tasks, discuss results, task design and methodological challenges. We highlight that almost all top performing systems utilize some sort of joint embedding for text-image retrieval and enable specification of temporal context in queries for known-item search. Whereas a combination of these techniques drive the currently top performing systems, we identify several future challenges for interactive video search engines and the Video Browser Showdown competition itself.

Assessing the Educational Quality of Training Videos for Collection of a Nasopharyngeal Swab

Introduction: The SARS-CoV-2 pandemic has forced healthcare systems to disseminate their training materials quickly and broadly, including instruction on identifying cases of infection through correct nasopharyngeal swabbing. Incorrect nasopharyngeal swabbing technique leads to substandard sampling, patient discomfort, and increased risk of complications. We set out to evaluate the quality of educational videos on the nasopharyngeal swab procedure. Methods: Using video search engines, videos on nasopharyngeal swabbing were identified and distributed to two reviewers. The quality of videos was assessed using a scoring system that examined indications, contraindications, personal protective equipment use, swab depth, swab angle, and audiovisual quality. Descriptive statistics and Spearman’s correlation coefficients were utilized to analyze video quality and its association with individual video characteristics. Results: Videos received an average composite score of 5.4 (range: 0-10), with about half of all videos properly discussing and demonstrating the nasopharyngeal swab technique. Over 62% of reviews indicated that the reviewer would not recommend the video to a trainee, with the vast majority identifying improper swab technique as the main factor. There were no statistically significant associations between video score and publication date, number of views, and subscribers to the publisher. Conclusion: Our study shows a glaring lack of quality educational videos on the nasopharyngeal swab procedure. Health care providers should be cautious when using educational videos to learn procedures as popularity may not be associated with accuracy.

VSRNet: End-to-end video segment retrieval with text query

Users are sometimes interested in specific segments of an untrimmed video when using the video search engine. Targeting at this demand, we explore a novel research topic of text query based video segment retrieval (VSR). Different from the conventional video retrieval task or localizing text descriptions in a single video, it requires the retrieval of the most relevant video from a large collection as well as localizing the start and end timestamps of a segment that matches the text query best from the video. A direct solution is to perform video-level matching first, and then apply description localization among such video candidates. Such two-stage based methods are not able to utilize complementary information of each stage, and are time-consuming in inference. In this paper, We propose VSRNet, an end-to-end framework that efficiently retrieves video at segment granularity with two branches. In the first branch, individual videos and texts are mapped to a common space for stand-alone ranking. In the second branch, we propose a supervised text-aligned attention mechanism and calculate the response of every frame to the text query, from which the frames with high scores are aggregated as segment proposals. Extensive experiments conducted on ActivityNet Captions and DiDeMo verify the effectiveness of our method and show that our solution significantly outperforms the state of the art.

Indexed Dataset from YouTube for a Content-Based Video Search Engine

Numerous researches on content-based video indexing and retrieval besides video search engines are tied to a large-scaled video dataset. Unfortunately, reduction in open-sourced datasets resulted in complications for novel approaches exploration. Although, video datasets that index video files located on public video streaming services have other purposes, such as annotation, learning, classification, and other computer vision areas, with little interest in indexing public video links for purpose of searching and retrieval. This paper introduces a novel large-scaled dataset based on YouTube video links to evaluate the proposed content-based video search engine, gathered 1088 videos, that represent more than 65 hours of video, 11,000 video shots, and 66,000 unmarked and marked keyframes, 80 different object names used for marking. Moreover, a state-of-the-art features vector, and combinational-based matching, beneficial to the accuracy, speed, and precision of the video retrieval process. Any video record in the dataset is represented by three features: temporal combination vector, object combination vector with shot annotations, and 6 keyframes, sideways with other metadata. Video classification for the dataset was also imposed to expand the efficiency of retrieval of video-based queries. A two-phased approach has been used based on object and event classification, storing video records in aggregations related to feature vectors extracted. While object aggregation stores video records with the maximal occurrence of extracted object/concept from all shots, event aggregation classify based on groups according to the number of shots per video. This study indexed 58 out of 80 different object/concept categories, each has 9 shot number groups.

The Association Between Health Information Seeking on the Internet and Physician Visits (The Seventh Tromsø Study - Part 4): Population-Based Questionnaire Study.

BackgroundThe internet is being widely used for seeking health information. However, there is no consensus on the association between health information seeking on the internet and the use of health care services.ObjectiveWe examined the association between health information seeking via the internet and physician visits. In addition, we investigated the association between online health information seeking and the decisions to visit and not to visit a physician.MethodsWe used the cross-sectional electronic health (eHealth) data of 18,197 participants from the seventh survey of the Tromsø Study (Tromsø 7). The participants were aged ≥40 years and living in Tromsø, Norway. We used logistic regression models to examine the association between online health information seeking and physician visits, the decision to visit a physician, and the decision not to visit a physician, with adjustment for the demographic status, socioeconomic status, and health status of the participants.ResultsThe use of Web search engines was associated with a physician visit. However, the association was moderated by age, and the OR decreased as age increased. The ORs for the use of Web search engines were 1.99 (95% CI 1.94-2.02) and 1.07 (95% CI 1.03-1.12) at ages 40 and 80 years, respectively. The decision to visit a physician was associated with the use of Web search engines (OR 2.95, 95% CI 2.03-4.46), video search engines (OR 1.43, 95% CI 1.21-1.70), and health apps (OR 1.26, 95% CI 1.13-1.42). The association between social media use and the decision to visit a physician was moderated by gender. Women who used social media had 1.42 (95% CI 1.31-1.55) times higher odds of deciding to visit a physician, whereas the decision to visit a physician was not different between men who used social media and those who did not use social media. Conversely, the decision not to visit a physician was associated with the use of Web search engines (OR 2.78, 95% CI 1.92-4.18), video search engines (OR 1.27, 95% CI 1.07-1.51), social media (OR 1.28, 95% CI 1.10-1.49), and health apps (OR 1.20, 95% CI 1.07-1.35).ConclusionsHealth information found on the internet was positively associated with both the decision to visit a physician and the decision not to visit a physician. However, the association of health information seeking with the decision to visit a physician was slightly stronger than the association with the decision not to visit a physician. This could imply that the use of eHealth services is associated with a resultant increase in physician visits. In summary, our findings suggest that the internet serves as a supplement to health care services rather than as a replacement.

Impact of the Use of Electronic Health Tools on the Psychological and Emotional Well-Being of Electronic Health Service Users (The Seventh Tromsø Study - Part 3): Population-Based Questionnaire Study.

BackgroundElectronic health (eHealth) has been described as a silver bullet for addressing how challenges of the current health care system may be solved by technological solutions in future strategies and visions for modern health care. However, the evidence of its effects on service quality and cost effectiveness remains unclear. In addition, patients’ psychological and emotional reactions to using eHealth tools are rarely addressed by the scientific literature.ObjectiveThis study aimed to assess how the psychological and emotional well-being of eHealth service users is affected by the use of eHealth tools.MethodsWe analyzed data from a population-based survey in Norway, conducted in the years 2015-2016 and representing 10,604 eHealth users aged over 40 years, to identify how the use of eHealth tools was associated with feeling anxious, confused, knowledgeable, or reassured. Associations between these four emotional outcomes and the use of four types of eHealth services (Web search engines, video search engines, health apps, and social media) were analyzed using logistic regression models.ResultsThe use of eHealth tools made 72.41% (6740/9308) of the participants feel more knowledgeable and 47.49% (4421/9308) of the participants feel more reassured about their health status. However, 25.69% (2392/9308) reported feeling more anxious and 27.88% (2595/9308) reported feeling more confused using eHealth tools. A high level of education and not having a full-time job were associated with positive reactions and emotions (feeling more knowledgeable and reassured), whereas low self-reported health status and not having enough friends who could provide help and support predicted negative reactions and emotions (ie, feeling anxious and confused). Overall, the positive emotional effects of eHealth use (feeling knowledgeable and reassured) were relatively more prevalent among users aged over 40 years than the negative emotional effects (ie, feeling anxious and confused). About one-fourth of eHealth users reported being more confused and anxious after using eHealth services.ConclusionsThe search for health information on the internet can be motivated by a range of factors and needs (not studied in this study), and people may experience a range of reactions and feelings following health information searching on the Web. Drawing on prior studies, we categorized reactions as positive and negative reactions. Some participants had negative reactions, which is challenging to resolve and should be taken into consideration by eHealth service providers when designing services (ie, including concrete information about how users can get more help and support). There is a need for more studies examining a greater range of reactions to online health information and factors that might predict negative reactions to health information on the Web.

Evaluation of surgical educational videos available for third year medical students

ABSTRACT Objectives: In this study we evaluated 40, top recommended, laparoscopic appendectomy and laparoscopic cholecystectomy videos located on public domain websites using eight criteria created by a panel of third year medical students and general surgeons. We hypothesized that there is a lack of quality, thorough educational laparoscopic surgical videos appropriate for third year medical students to review in preparation for the Surgery rotation. Methods: Utilizing a panel, which included four third year medical students and two general surgeons, we created an ‘ideal medical student educational video checklist.’ This checklist included 8 vital criteria. We selected 40, top recommended, videos available on YouTube and Google Video search engines, using ‘laparoscopic cholecystectomy’ and ‘laparoscopic appendectomy’ as key terms. Each video was evaluated by four third year medical students individually, using a binary system ‘meets’ or ‘does not meet’ each criterion. Individual scores were averaged, producing a single score for each video. Results: 0/40 (0%) of the videos met all eight of the criteria. 26/40 (65%) of the videos did not meet half of the criteria. The top performing videos 7/40 (17%) only met 5/8 criteria. Conclusions: We identified a lack of quality and thorough educational surgical videos appropriate for third year medical students and a need for improved online video based instruction. Our checklist can be utilized as a guide for anyone creating surgical videos for medical student education in the future.

Survey on Semantic Based Video Hierarchy for Browsing Large Video Repositories

The number of videos exploding on the web is a good challenge in automatic video search. Existing Video search engines, like YouTube, imagine the sorted list of search results. The straightforward list system may be suitable for finding an ad or a music video, although queries cannot be handled with complex header systems, so users create difficulties to find the necessary information. During this event, the best way to a semi-structured array in the nodes that covers all the main headings. Obviously, each question manually changes headlines and thanks for a good range of search titles. By implementing video descriptions on their linguistic information page, we use a prototype-based streamline cluster regime. We will take this information from linguistic information to the complete linguistic information we make by utilizing a sub-ontology conception in our site.

Automatic Evaluation of Video Search engines in Persian Web domain based on Majority Voting

Automatic Evaluation of Video Search engines in Persian Web domain based on Majority Voting

An Accurate Efficient and Scalable Event Based Video Search Method Using Spectral Clustering

Web mining discovers enormous set of data and gets hidden and valuable information which contains text, images, audio and video files from the web search engine which is software that provides a significant result of information. Video rehabilitation for the context gives efficient comprehension of the video content. Video retrieval refers to the task of retrieving most relevant videos from the video Search engine but the outcome listed result could not achieve the relevant videos according to the user needs. This paper addresses Event based Video Retrieval (EBVR) uses metadata, which gives the accurate result. The aim is detect the circumstances of a focal point such as birthday party. In order to overcome this issue, we proposed a personalization approach which captures the user query relevance to their event. Video preprocessing method used to extract related precision data and spectral clustering technique for Video Categorization which yields event extraction and contributes associated video.

Semantic-Based Video Retrieval Survey

There is a tremendous growth of digital data due to the stunning progress of digital devices which facilitates capturing them. Digital data include image, text, and video. Video represents a rich source of information. Thus, there is an urgent need to retrieve, organize, and automate videos. Video retrieval is a vital process in multimedia applications such as video search engines, digital museums, and video-on-demand broadcasting. In this paper, the different approaches of video retrieval are outlined and briefly categorized. Moreover, the different methods that bridge the semantic gap in video retrieval are discussed in more details.

Parallelizing Multi-featured Content Based Search and Retrieval of Videos through High Performance Computing

<p>Video Retrieval is an important technology that helps to design video search engines and allow users to browse and retrieve videos of interest from huge databases. Though, there are many existing techniques to search and retrieve videos based on spatial and temporal features but are unable to perform well resulting in high ranking of irrelevant videos leading to poor user satisfaction. In this paper an efficient multi-featured method for matching and extraction is proposed in parallel paradigm to retrieve videos accurately and quickly from the collection. Proposed system is tested on datasets that contains various categories of videos of varying length such as traffic, sports, nature etc. Experimental results show that around 80% of accuracy is achieved in searching and retrieving video. Through the use of high performance computing, the parallel execution performs 5 times faster in locating and retrieving videos of intrest than the sequential execution.</p>

Analysis of Search Engine Optimization Algorithm

SEO stands for engine optimization. It is the process of getting traffic from the free, organic, editorial or natural search results on search engines. The earlier (or higher ranked on the search results page), and more frequently a site appears in the search results list, the more visitors it will receive from the search engine's users, and these visitors can be converted into customers. SEO may target different kinds of search, including image search, local search, video search, academic search, news search and industry-specific vertical search engines. As an Internet marketing strategy, SEO considers how search engines work, what people search for, the actual search terms or keywords typed into search engines and which search engines are preferred by their targeted audience. Optimizing a website may involve editing its content, HTML and associated coding to both increase its relevance to specific keywords and to remove barriers to the indexing activities of search engines. Promoting a site to increase the number of backlinks, or inbound links, is another SEO tactic. As of May 2015, mobile search has finally surpassed desktop search, Google is developing and pushing mobile search as the future in all of its products and many brands are beginning to take a different approach on their internet strategies.

Action and Event Recognition in Videos by Learning From Heterogeneous Web Sources.

In this paper, we propose new approaches for action and event recognition by leveraging a large number of freely available Web videos (e.g., from Flickr video search engine) and Web images (e.g., from Bing and Google image search engines). We address this problem by formulating it as a new multi-domain adaptation problem, in which heterogeneous Web sources are provided. Specifically, we are given different types of visual features (e.g., the DeCAF features from Bing/Google images and the trajectory-based features from Flickr videos) from heterogeneous source domains and all types of visual features from the target domain. Considering the target domain is more relevant to some source domains, we propose a new approach named multi-domain adaptation with heterogeneous sources (MDA-HS) to effectively make use of the heterogeneous sources. In MDA-HS, we simultaneously seek for the optimal weights of multiple source domains, infer the labels of target domain samples, and learn an optimal target classifier. Moreover, as textual descriptions are often available for both Web videos and images, we propose a novel approach called MDA-HS using privileged information (MDA-HS+) to effectively incorporate the valuable textual information into our MDA-HS method, based on the recent learning using privileged information paradigm. MDA-HS+ can be further extended by using a new elastic-net-like regularization. We solve our MDA-HS and MDA-HS+ methods by using the cutting-plane algorithm, in which a multiple kernel learning problem is derived and solved. Extensive experiments on three benchmark data sets demonstrate that our proposed approaches are effective for action and event recognition without requiring any labeled samples from the target domain.

Automatic and online setting of similarity thresholds in content-based visual information retrieval problems

Several information recovery systems use functions to determine similarity among objects in a collection. Such functions require a similarity threshold, from which it becomes possible to decide on the similarity between two given objects. Thus, depending on its value, the results returned by systems in a search may be satisfactory or not. However, the definition of similarity thresholds is difficult because it depends on several factors. Typically, specialists fix a threshold value for a given system, which is used in all searches. However, an expert-defined value is quite costly and not always possible. Therefore, this study proposes an approach for automatic and online estimation of the similarity threshold value, to be specifically used by content-based visual information retrieval system (image and video) search engines. The experimental results obtained with the proposed approach prove rather promising. For example, for one of the case studies, the performance of the proposed approach achieved 99.5 % efficiency in comparison with that obtained by a specialist using an empirical similarity threshold. Moreover, such automated approach becomes more scalable and less costly.

Gaze movement-driven random forests for query clustering in automatic video annotation

In the recent years, the rapid increase of the volume of multimedia content has led to the development of several automatic annotation approaches. In parallel, the high availability of large amounts of user interaction data, revealed the need for developing automatic annotation techniques that exploit the implicit user feedback during interactive multimedia retrieval tasks. In this context, this paper proposes a method for automatic video annotation by exploiting implicit user feedback during interactive video retrieval, as this is expressed with gaze movements, mouse clicks and queries submitted to a content-based video search engine. We exploit this interaction data to represent video shots with feature vectors based on aggregated gaze movements. This information is used to train a classifier that can identify shots of interest for new users. Subsequently, we propose a framework that during testing: a) identifies topics (expressed by query clusters), for which new users are searching for, based on a novel clustering algorithm and b) associates multimedia data (i.e., video shots) to the identified topics using supervised classification. The novel clustering algorithm is based on random forests and is driven by two factors: first, by the distance measures between different sets of queries and second by the homogeneity of the shots viewed during each query cluster defined by the clustering procedure; this homogeneity is inferred from the performance of the gaze-based classifier on these shots. The evaluation shows that the use of aggregated gaze data can be exploited for video annotation purposes.

Invited Paper] Strategies for Searching Video Content with Text Queries or Video Examples

The large number of user-generated videos uploaded on to the Internet everyday has led to many commercial video search engines, which mainly rely on text metadata for search. However, metadata is often lacking for user-generated videos, thus these videos are unsearchable by current search engines. Therefore, content-based video retrieval (CBVR) tackles this metadata-scarcity problem by directly analyzing the visual and audio streams of each video. CBVR encompasses multiple research topics, including low-level feature design, feature fusion, semantic detector training and video search/reranking. We present novel strategies in these topics to enhance CBVR in both accuracy and speed under different query inputs, including pure textual queries and query by video examples. Our proposed strategies have been incorporated into our submission for the TRECVID 2014 Multimedia Event Detection evaluation, where our system outperformed other submissions in both text queries and video example queries, thus demonstrating the effectiveness of our proposed approaches.

Atoms and Bits: Rethinking Manufacturing

An Interview with Neil GershenfeldNeil Gershenfeld talks with Jim Euchner about the Internet of Things and the coming revolution in manufacturing.Neil Gershenfeld is a pioneer in understanding the boundary between the digital and physical worlds-how objects become data in the network and how data in the network become objects. The technology from his lab, MIT's Center for Bits and Atoms, has been seen and used in a wide variety of settings, from the Museum of Modern Art in New York and the World Economic Forum to rural Indian villages and innercity community centers. In this interview, he discusses the true nature of the Internet of Things, how distributed fab labs can enable anyone to make almost anything, the power of global knowledge sharing with innovation at the nodes, and the future of manufacturing using digital materials.JIM EUCHNER [JE]: The Internet of Things has become quite a buzzword. I think you define it more precisely than some do. Can you give your definition-what is the Internet of Things and what is it not?NEIL GERSHENFELD [NG]: Sure. I was part of a group that started what became called the Internet of Things. A decade or so ago, a group at MIT was working on embedding Internet protocols into devices and showing that, with a few hundred bytes of code and a dollar a part, you could implement a minimal IP stack. That meant that individual devices could talk to the Internet. So the simplest statement about the Internet of Things is that it means IP embedded in devices.A lot of the things called Internet of Things today are really BITNET of Things, and they're being done by people who don't understand the difference between the BITNET and the Internet. They are ignorant of the underlying architecture. There are a few core architectural principles that make the Internet work that you can apply to embedded devices. After 10 or so years, we've had to reconnect the future with the past concerning those architectural principles.BITNET was a connection of mainframes, and it was a very high barrier to entry. To expand the BITNET, you needed to add another mainframe, and that was a large investment that was centrally managed. Users used terminals connected to the mainframes. It was similar to the old phone system: what a telephone did was defined not by the phone but by the central office switch. Adding a new central office switch was a big investment. One of the core principles that led to the Internet was the concept of what was called end-to-end, which roughly means that what the Internet does is defined by what you connect to it.The Internet transports packets; it uses a common representation for those packets. At the time it was emerging, there were many dissimilar networks, and the Internet settled on a common representation that could travel across them. But the Internet is really defined by what you connect to it, not by its physical construction. The Internet could grow by adding things to the edges, not by changing what was at the center. That enabled people to invent instant messaging and VoIP and video streaming and search engines, not by changing the architecture of the Internet but by innovating at the edges.Now the reason so many of the things called Internet of Things are actually the BITNET of Things is that people think in terms of devices, like a light sensor or a thermostat or a switch, connected to a server that proxies its state to build an application. Manipulation of the sensor can be done through the Internet, but the state of the switch or the light is not contained in the switch or light, it's contained in the remote server. That means that to change the function of the device, you have to change the remote thing, not the local thing. You can't innovate at the edges because you need access to the central controller.To be the Internet of Things in my definition, the light or the switch has to be a full-fledged citizen of the Internet. …