Video Database Research Articles

Oppositional Grass Hopper Optimization with Fuzzy Classifier for Face Recognition from Video Database

Oppositional Grass Hopper Optimization with Fuzzy Classifier for Face Recognition from Video Database

Perspective on Emotion Detection for Automotive Applications: Performance Evaluation of Two Emotion AI SDKs

The work presented herein quantifies the limitations of the technology provided by two prominent suppliers in Emotion AI. Each Software Development Kit (SDK) performance was measured for accuracy using image and video databases. The results indicate that while the SDKs show high accuracy in detecting positive emotions (e.g., Happy), the performance suffered for negative emotions (e.g., Angry) due to missed and false detections. The results were worse for structured video datasets and degraded further when subjects were in naturalistic settings. Although Emotion AI have improved greatly in recent years, the current versions are not reliable enough for automotive applications. The paper provides perspectives on the reasons for subpar performance and guidance for improvement for future emotion estimation software.

Depth Perception Assessment of 3D Videos Based on Stereoscopic and Spatial Orientation Structural Features

Depth quality of stereoscopic three-dimensional (S3D) videos is a significant factor which directly affects the quality of experience (QoE) associated with 3D video applications and services. Nevertheless, there remain limited reports on the investigation of depth perception and depth quality evaluation of S3D videos, which impedes further advancement and deployment of 3D video technology. This paper reports a series of subjective experiments which have been conducted to investigate the depth perception and its related properties of the human visual system (HVS) using S3D video compressed by the H.264/AVC standard. The experimental results reveal that the HVS response in depth perception varies at different frequencies and in varying orientations, and the distortions introduced by video coding can cause the loss of and/or variation in depth perception. By integration of binocular and monocular features (BM) extracted from left and right views of S3D video with respect to depth perception, a depth quality assessment model, herein referred to as BM-DQAM, is devised by training these stereoscopic and spatial orientation structural features with a support vector regression model. It is shown that the BM-DQAM provides a novel no-reference metric for evaluation of the depth quality of S3D videos. Based on two publicly available 3D video databases and the proposed depth perception assessment database, the experimental results show that the BM-DQAM has demonstrated better performance in assessing the depth quality in S3D video viewing than that of other metrics reported in the published literatures, correlating well with the HVS response in the depth perception assessment experiment.

CAM-FRN: Class Attention Map-Based Flare Removal Network in Frontal-Viewing Camera Images of Vehicles

In recent years, active research has been conducted on computer vision and artificial intelligence (AI) for autonomous driving to increase the understanding of the importance of object detection technology using a frontal-viewing camera. However, using an RGB camera as a frontal-viewing camera can generate lens flare artifacts due to strong light sources, components of the camera lens, and foreign substances, which damage the images, making the shape of objects in the images unrecognizable. Furthermore, the object detection performance is significantly reduced owing to a lens flare during semantic segmentation performed for autonomous driving. Flare artifacts pose challenges in their removal, as they are caused by various scattering and reflection effects. The state-of-the-art methods using general scene image retain artifactual noises and fail to eliminate flare entirely when there exist severe levels of flare in the input image. In addition, no study has been conducted to solve these problems in the field of semantic segmentation for autonomous driving. Therefore, this study proposed a novel lens flare removal technique based on a class attention map-based flare removal network (CAM-FRN) and a semantic segmentation method using the images in which the lens flare is removed. CAM-FRN is a generative-based flare removal network that estimates flare regions, generates highlighted images as input, and incorporates the estimated regions into the loss function for successful artifact reconstruction and comprehensive flare removal. We synthesized a lens flare using the Cambridge-driving Labeled Video Database (CamVid) and Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago (KITTI) datasets, which are road scene open datasets. The experimental results showed that semantic segmentation accuracy in images with lens flare was removed based on CAM-FRN, exhibiting 71.26% and 60.27% mean intersection over union (mIoU) in the CamVid and KITTI databases, respectively. This indicates that the proposed method is significantly better than state-of-the-art methods.

Form follows function: Applying photographic content analysis to forensic firearm identification.

Drawing forensic conclusions from an image or a video is known as "photographic content analysis." It involves the analysis of an image, as well as objects, actions, and events depicted in images or video. In recent years, photographic depictions of objects suspected as illegal firearms have substantially increased, appearing on CCTV surveillance footage, captured by mobile phones and shared on social media. However, the law in Israel states that a person can be charged with illegally possessing a firearm only if it can be proven that the object is capable of shooting with lethal bullet energy. This becomes more challenging in cases where the firearm was not physically seized, and the evidence exclusively consists of images and video. In this study, photographic content analysis was applied to images and video where objects suspected as commercial or improvised firearms had been depicted. An image and event sequence reconstruction video databases of both firearms and replicas were created in order to better define firearm-specific functional morphological features. We demonstrate that it is possible to classify an object as a firearm by analyzing the functional, and not only the esthetic, morphology in images and video. It is also shown that event sequence reconstruction in video may be used to infer that an object suspected as a firearm has the capacity to shoot by confirming the occurrence of a shooting act or shooting process. Thus, photographic content analysis may be used to forensically establish that an object depicted in an image or a video is a firearm by ruling out other known scenarios, and without physically seizing it.

Video-Based versus On-Site Neonatal Pain Assessment in Neonatal Intensive Care Units: The Impact of Video-Based Neonatal Pain Assessment in Real-World Scenario on Pain Diagnosis and Its Artificial Intelligence Application.

Neonatal pain assessment (NPA) represents a huge global problem of essential importance, as a timely and accurate assessment of neonatal pain is indispensable for implementing pain management. To investigate the consistency of pain scores derived through video-based NPA (VB-NPA) and on-site NPA (OS-NPA), providing the scientific foundation and feasibility of adopting VB-NPA results in a real-world scenario as the gold standard for neonatal pain in clinical studies and labels for artificial intelligence (AI)-based NPA (AI-NPA) applications. A total of 598 neonates were recruited from a pediatric hospital in China. This observational study recorded 598 neonates who underwent one of 10 painful procedures, including arterial blood sampling, heel blood sampling, fingertip blood sampling, intravenous injection, subcutaneous injection, peripheral intravenous cannulation, nasopharyngeal suctioning, retention enema, adhesive removal, and wound dressing. Two experienced nurses performed OS-NPA and VB-NPA at a 10-day interval through double-blind scoring using the Neonatal Infant Pain Scale to evaluate the pain level of the neonates. Intra-rater and inter-rater reliability were calculated and analyzed, and a paired samples t-test was used to explore the bias and consistency of the assessors' pain scores derived through OS-NPA and VB-NPA. The impact of different label sources was evaluated using three state-of-the-art AI methods trained with labels given by OS-NPA and VB-NPA, respectively. The intra-rater reliability of the same assessor was 0.976-0.983 across different times, as measured by the intraclass correlation coefficient. The inter-rater reliability was 0.983 for single measures and 0.992 for average measures. No significant differences were observed between the OS-NPA scores and the assessment of an independent VB-NPA assessor. The different label sources only caused a limited accuracy loss of 0.022-0.044 for the three AI methods. VB-NPA in a real-world scenario is an effective way to assess neonatal pain due to its high intra-rater and inter-rater reliability compared to OS-NPA and could be used for the labeling of large-scale NPA video databases for clinical studies and AI training.

Screen content video quality assessment based on spatiotemporal sparse feature

In recent years, the explosive growth of application scenarios have occurred in screen content videos (SCVs), in which the SCVs are unavoidably suffered from the quality degradation and the video quality assessment (VQA) of the SCVs becomes very essential. In view of this, a full-reference VQA algorithm, called the spatiotemporal sparse feature-based model (SSFM) is proposed in this paper, aiming to give a precise and efficient quality evaluation about the distorted SCVs. Note that the SCVs are full of edge information which the human eyes are highly sensitive to, and the sparse coding can provide accurate quantitative predictions which are consistent with the perception resulted from the cerebral cortex in various receptive field models of the visual cortex. With these considerations, three dimensional Difference of Gaussian (3D-DOG) filter and 3D Sparse Dictionary are developed to extract multi-scale spatiotemporal features and obtain spatiotemporal sparse features respectively, from the reference and distorted SCVs. Based on these features, the spatiotemporal sparse feature similarity can be measured and followed by generating the quality scores of the distorted SCVs under evaluation. Compared to other classic and state-of-the-art image/video quality evaluation metrics, the experimental results of the proposed SSFM on the screen content video database (SCVD) are more consistent with the perceived evaluation of SCVs according to the human visual system (HVS).

DoveDB: A Declarative and Low-Latency Video Database

Concerning the usability and efficiency to manage video data generated from large-scale cameras, we demonstrate DoveDB, a declarative and low-latency video database. We devise a more comprehensive video query language called VMQL to improve the expressiveness of previous SQL-like languages, which are augmented with functionalities for model-oriented management and deployment. We also propose a light-weight ingestion scheme to extract tracklets of all the moving objects and build semantic indexes to facilitate efficient query processing. For user interaction, we construct a simulation environment with 120 cameras deployed in a road network and demonstrate three interesting scenarios. Using VMQL, users are allowed to 1) train a visual model using SQL-like statement and deploy it on dozens of target cameras simultaneously for online inference; 2) submit multi-object tracking (MOT) requests on target cameras, store the ingested results and build semantic indexes; and 3) issue an aggregation or top- k query on the ingested cameras and obtain the response within milliseconds. A preliminary video introduction of DoveDB is available at https://www.youtube.com/watch?v=N139dEyvAJk

EQUI-VOCAL Demonstration: Synthesizing Video Queries from User Interactions

We demonstrate EQUI-VOCAL, a system that synthesizes compositional queries over videos from user feedback. EQUI-VOCAL enables users to query a video database for complex events by providing a few positive and negative examples of what they are looking for and labeling a small number of additional system-selected examples. Using those user inputs, EQUI-VOCAL synthesizes declarative queries that can then retrieve additional instances of the desired events. The demonstration makes two contributions: it introduces EQUI-VOCAL's graphical user interface and enables conference attendees to experiment with EQUI-VOCAL on a variety of queries. Both enable users to gain a better understanding of EQUI-VOCAL's query synthesis approach and to explore the impact of hyperparameters and label noise on system performance.

DeepVQL: Deep Video Queries on PostgreSQL

The recent development of mobile and camera devices has led to the generation, sharing, and usage of massive amounts of video data. As a result, deep learning technology has gained attention as an alternative for video recognition and situation judgment. Recently, new systems supporting SQL-like declarative query languages have emerged, focusing on developing their own systems to support new queries combined with deep learning that are not supported by existing systems. The proposed DeepVQL system in this paper is implemented by expanding the PostgreSQL system. DeepVQL supports video database functions and provides various user-defined functions for object detection, object tracking, and video analytics queries. The advantage of this system is its ability to utilize queries with specific spatial regions or temporal durations as conditions for analyzing moving objects in traffic videos.

Forged facial video detection framework based on multi-region temporal relationship feature

Face generation and manipulation techniques based on deep learning have enabled the creation of sophisticated forged facial videos, which are indistinguishable by human eyes. However, the illegal use of deep fake technology will have a serious impact on social stability, personal reputation, and even national security. Therefore, the detection technology of fake facial videos is of great significance to protect national security and maintain social order. Although the existing video-based fake face video detection technology has achieved good detection performance on the public fake face video database, there are still the following problems: (1) the existing technology uses a 2D attention mechanism to obtain local region features from face images and lacks a 3D attention mechanism to obtain local area features from face videos; (2) after obtaining local area features, the existing technology is directly used to classify or only model the inter-regional relationship of images without modeling the temporal relationship between regions of the video. This paper proposes a fake facial video detection framework based on multi-region temporal relationship features, including designing a three-dimensional attention mechanism to extract local features of multiple regions of the face from the video. In order to model the time series relationship between different face areas, a time series graph convolution neural network is also introduced to extract the time series relationship features between multiple areas. In order to model the time sequence relationship between different face regions, the convolution neural network of a time sequence diagram is also introduced to extract the characteristics of the time sequence relationship between multiple regions. Through the change characteristics of the time sequence relationship between face regions, the timing inconsistency of the face video is detected so as to determine whether the face has been deeply forged. Through experiments on multiple datasets, the experimental results of the model accuracy test show that the method proposed in the present invention achieves the highest detection accuracy, and the accuracy in the FaceForensics++ (low definition) dataset is 18.19% higher than that of the benchmark method. The experimental results of the generalization ability test show that the method proposed in the present invention achieves the highest generalization performance, and the detection accuracy in the Celeb-DF dataset is 11.92% higher than that of the benchmark method.

Formation and Assessment of a Laryngology Pathology Video Atlas for Resident Education

Conventional reference images of laryngeal pathologies may provide educational value for Otolaryngology-Head & Neck Surgery (OHNS) residents, but observing dynamic vocal fold function is critical for diagnosis. Our aim was to develop and validate a video atlas of laryngeal pathologies for resident education in OHNS. A multi-institution, prospective case-control study. Ten videos showing 10 representative laryngeal pathologies were verified by two laryngologists. Six videos per category with kappa>0.8 were included in the video database. A collection of the videos was shown to a group of OHNS residents in a quiz fashion to determine if senior trainees would score higher than junior trainees. Another group of residents in OHNS was recruited and randomized to control or intervention. The control group was shown a quiz of 10 laryngeal videos at baseline and 24 weeks later. The intervention group was shown quizzes at baseline and every 6 weeks, ending at 24 weeks. Free-text diagnoses were scored for accuracy. Descriptive statistics, two-tailed tests, and analysis of covariance were performed. Twenty-nine residents participated, with 14 (48.3%) randomized to control, and 15 (51.7%) to the intervention. The postgraduateyear (PGY) level had a significant impact on diagnostic performance. PGY1 and 2 had a significantly lower score than PGY5 (P=0.017 and P=0.035, respectively). PGY3 and PGY4 scores were not statistically different from PGY5 scores. The mean score difference between groups decreases as the PGY level increases (mean difference between groups=0.87, P=0.153), but this was not significant. The current study has created a validated collection of videos that are representative of common laryngeal pathologies and can be easily incorporated into resident video-based learning. Future directions include larger multi-site studies to further elucidate whether repeated viewing of this video atlas can improve OHNS resident laryngology knowledge.

Mobile Videography on YouTube and the Significance of the Marginalised Creator

This research aims to identify the variables that contribute to the differential viewership of mobile-phone videos on YouTube. To achieve this, a Generalized Linear Mixed Model (GLMM) is employed to examine the correlations among various variables in a randomly selected video database. The findings of the analysis reveal several significant and culturally insightful associations. Notably, subgenres such as melodrama (soap-operas or telenovelas), action, and comedy emerge as distinctive categories, irrespective of the total number of views. Comedies and action videos created by male filmmakers receive higher viewership. Similarly, melodramas created by female filmmakers, particularly those hailing from marginalized and rural backgrounds, hold greater statistical significance. These findings highlight the relevance of gender and social context in shaping viewer preferences and challenge the conventions of mainstream media in the respective countries of origin.

Can artificial intelligence help decision-making in arthroscopy? Part 1: Use of a standardized analysis protocol improves inter-observer agreement of arthroscopic diagnostic assessments of the long head of biceps tendon in small rotator cuff tears

IntroductionInjuries of the long head of biceps (LHB) tendon are common but difficult to diagnose clinically or using imaging. Arthroscopy is the preferred means of diagnostic assessment of the LHB, but it often proves challenging. Its reliability and reproducibility have not yet been assessed. Artificial intelligence (AI) could assist in the arthroscopic analysis of the LHB. The main objective of this study was to evaluate the inter-observer agreement for the specific LHB assessment, according to an analysis protocol based on images of interest. The secondary objective was to define a video database, called “ground truth”, intended to create and train AI for the LHB assessment. HypothesisThe hypothesis was that the inter-observer agreement analysis, on standardized images, was strong enough to allow the “ground truth” videos to be used as an input database for an AI solution to be used in making arthroscopic LHB diagnoses. Materials and methodOne hundred and ninety-nine sets of standardized arthroscopic images of LHB exploration were evaluated by 3 independent observers. Each had to characterize the healthy or pathological state of the tendon, specifying the type of lesion: partial tear, hourglass hypertrophy, instability, fissure, superior labral anterior posterior lesion (SLAP 2), chondral print and pathological pulley without instability. Inter-observer agreement levels were measured using Cohen's Kappa (K) coefficient and Kappa Accuracy. ResultsThe strength of agreement was moderate to strong according to the observers (Kappa 0.54 to 0.7 and KappaAcc from 86 to 92%), when determining the healthy or pathological state of the LHB. When the tendon was pathological, the strength of agreement was moderate to strong when it came to a partial tear (Kappa 0.49 to 0.71 and KappaAcc from 85 to 92%), fissure (Kappa −0.5 to 0.7 and KappaAcc from 36 to 93%) or a SLAP tear (0.54 to 0.88 and KappaAcc from 90 to 97%). It was low for unstable lesion (Kappa 0.04 to 0.25 and KappaAcc from 36 to 88%). ConclusionThe analysis of the LHB, from arthroscopic images, had a high level of agreement for the diagnosis of its healthy or pathological nature. However, the agreement rate decreased for the diagnosis of rare or dynamic tendon lesions. Thus, AI engineered from human analysis would have the same difficulties if it was limited only to an arthroscopic analysis. The integration of clinical and paraclinical data is necessary to improve the arthroscopic diagnosis of LHB injuries. It also seems to be an essential prerequisite for making a so-called “ground truth” database for building a high-performance AI solution. Level of evidenceIII; inter-observer prospective series.

Multiview Monitoring of Individual Cattle Behavior Based on Action Recognition in Closed Barns Using Deep Learning.

Cattle behavior recognition is essential for monitoring their health and welfare. Existing techniques for behavior recognition in closed barns typically rely on direct observation to detect changes using wearable devices or surveillance cameras. While promising progress has been made in this field, monitoring individual cattle, especially those with similar visual characteristics, remains challenging due to numerous factors such as occlusion, scale variations, and pose changes. Accurate and consistent individual identification over time is therefore essential to overcome these challenges. To address this issue, this paper introduces an approach for multiview monitoring of individual cattle behavior based on action recognition using video data. The proposed system takes an image sequence as input and utilizes a detector to identify hierarchical actions categorized as part and individual actions. These regions of interest are then inputted into a tracking and identification mechanism, enabling the system to continuously track each individual in the scene and assign them a unique identification number. By implementing this approach, cattle behavior is continuously monitored, and statistical analysis is conducted to assess changes in behavior in the time domain. The effectiveness of the proposed framework is demonstrated through quantitative and qualitative experimental results obtained from our Hanwoo cattle video database. Overall, this study tackles the challenges encountered in real farm indoor scenarios, capturing spatiotemporal information and enabling automatic recognition of cattle behavior for precision livestock farming.

Deep Learning Model for Automated Trainee Assessment During High-Fidelity Simulation.

Implementation of competency-based medical education has necessitated more frequent trainee assessments. Use of simulation as an assessment tool is limited by access to trained examiners, cost, and concerns with interrater reliability. Developing an automated tool for pass/fail assessment of trainees in simulation could improve accessibility and quality assurance of assessments. This study aimed to develop an automated assessment model using deep learning techniques to assess performance of anesthesiology trainees in a simulated critical event. The authors retrospectively analyzed anaphylaxis simulation videos to train and validate a deep learning model. They used an anaphylactic shock simulation video database from an established simulation curriculum, integrating a convenience sample of 52 usable videos. The core part of the model, developed between July 2019 and July 2020, is a bidirectional transformer encoder. The main outcome was the F1 score, accuracy, recall, and precision of the automated assessment model in analyzing pass/fail of trainees in simulation videos. Five models were developed and evaluated. The strongest model was model 1 with an accuracy of 71% and an F1 score of 0.68. The authors demonstrated the feasibility of developing a deep learning model from a simulation database that can be used for automated assessment of medical trainees in a simulated anaphylaxis scenario. The important next steps are to (1) integrate a larger simulation dataset to improve the accuracy of the model; (2) assess the accuracy of the model on alternative anaphylaxis simulations, additional medical disciplines, and alternative medical education evaluation modalities; and (3) gather feedback from education leadership and clinician educators surrounding the perceived strengths and weaknesses of deep learning models for simulation assessment. Overall, this novel approach for performance prediction has broad implications in medical education and assessment.

Multimodality Video Acquisition System for the Assessment of Vital Distress in Children.

In children, vital distress events, particularly respiratory, go unrecognized. To develop a standard model for automated assessment of vital distress in children, we aimed to construct a prospective high-quality video database for critically ill children in a pediatric intensive care unit (PICU) setting. The videos were acquired automatically through a secure web application with an application programming interface (API). The purpose of this article is to describe the data acquisition process from each PICU room to the research electronic database. Using an Azure Kinect DK and a Flir Lepton 3.5 LWIR attached to a Jetson Xavier NX board and the network architecture of our PICU, we have implemented an ongoing high-fidelity prospectively collected video database for research, monitoring, and diagnostic purposes. This infrastructure offers the opportunity to develop algorithms (including computational models) to quantify vital distress in order to evaluate vital distress events. More than 290 RGB, thermographic, and point cloud videos of each 30 s have been recorded in the database. Each recording is linked to the patient's numerical phenotype, i.e., the electronic medical health record and high-resolution medical database of our research center. The ultimate goal is to develop and validate algorithms to detect vital distress in real time, both for inpatient care and outpatient management.

A novel hybrid feature extraction and ensemble C3D classification for anomaly detection in surveillance videos

Anomaly detection in several deep learning frameworks are recently presented on real-time video databases as a challenging task. However, these frameworks have high false positive rate (FPR) and error rate due to various backgrounds, motion appearance and semantic high-level and low-level features for anomaly detection through action classification. Also, extraction of features and classification are the major problems in traditional convolution neural network (CNN) on real-time video databases. The proposed work is a novel action classification framework which is designed and implemented on large video databases with high true positive rate (TPR) and error rate. In this framework, Kalman based incremental principal component analysis (IPCA) feature extraction method; C3D and non-linear support vector machine (SVM) classifier are used to improve the action prediction (anomaly detection) on the large real-time video databases. The proposed frame work shown new results of high computation performance than the traditional deep learning frameworks for action classification.

Categorization Format for YouTube Medical Educational Videos: An Experience with Embryology

Background: Users of YouTube videos face considerable difficulty in handling the rapidly growing uploaded material. Videos can fall into various genres, and a single video can possess a number of characteristics. Rationale- Customized selection of the most appropriate videos for learning, teaching or research is hard to achieve. Appropriate tagging of videos for different characteristics by the uploaders for helping audience in their selection warrant meticulously designed structured format for categorizing the videos based on multiple characteristics. But no complete structured format is available for such categorization. Material & Methods: A cross-sectional analysis of 120 YouTube medical educational videos on Embryology selected through a multiphase systematic method using search terms chosen from selected focus areas of Embryology was done. This was aimed at identifying video characteristics of four functional natures: basic, academic, technical and esthetic. Using the ‘identified’ and related ‘identifiable’ characteristics, a “Proposed categorization format” was developed. Feedback on this document was obtained from five stakeholder groups i.e., anatomists, medical educationists, medical video creators, film/video/graphics experts and postgraduate students of Anatomy through a multiple-group discussion session. Results: Consensuses built on their opinions and lone voices were incorporated to develop the “Final version” of the format that possesses 38 Broad Characteristics, 179 Options, 59 Suboptions and fourteen Sub-suboptions. Conclusion: This format should be helpful in establishing a comprehensive database of medical educational videos on Embryology and with slight customization, on other anatomical subdivisions and other disciplines. But before that validation of the format by putting them into use should be justified.

Toward improved surveillance of Aedes aegypti breeding grounds through artificially augmented data

The Aedes aegypti mosquito transmits several diseases, including dengue, zika, and chikungunya. To prevent these diseases, identifying and removing mosquito breeding sites is essential, but it is a time-consuming and labor-intensive task. To improve efficiency, computer vision and machine learning can be used to detect potential breeding grounds automatically. In this context, we explore the use of data augmentation strategies including random scaling, rotation, as well as color and brightness adjustments for improving the automatic detection of potential Aedes aegypti breeding grounds using videos acquired by a drone. The faster region-based convolutional neural network (Faster R-CNN) and the you only look once (YOLO) v5 object detectors are used on a database of aerial videos containing breeding-related objects. When employing the data augmentation, tire-detection results show F1 scores of 0.79 and 0.81 for the Faster R-CNN and YOLOv5 networks, respectively, surpassing current state-of-the-art values. The detection performance of the algorithms increased by up to 14.1%, which is a significant improvement. These results indicate that artificial data augmentation reduces overfitting, improving the models’ robustness. The developed system can be employed to help health agencies in locating potential Aedes aegypti outbreaks more efficiently.