Video Description Research Articles

Exploring the Spatio‐Temporal Aware Graph for video captioning

Describing video content automatically in natural language sentences is a long-standing challenge in computer vision. Although existing methods that capture relational information among objects have made significant strides in the past years, the detailed geometrical and temporal information of objects remains to be further explored. To address this problem, a novel Spatio-Temporal Aware Graph is proposed to capture more elaborate visual representations, which are able to exploit the detailed spatio-temporal clues of the extracted object features. By performing graph-structured aggregation, the proposed model is capable of capturing not only the interactions among objects but also the detailed spatio-temporal relations. Meanwhile, a Frame Similarity Graph is constructed on frame features to learn comprehensive representations, which aim to extract the global information that the object feature lacks. Moreover, to capture rich video semantics from different perspectives, multiple video representations, that is appearance and motion information, are utilised to learn discriminative representations. Experiments on the prevalent benchmarks: Microsoft Video Description Corpus and Microsoft Research Video to Text demonstrate that the proposed approach achieves state-of-the-art performance in several widely used evaluation metrics: BLEU-4, METEOR, ROUGE, and CIDEr.

GssMILP for anomaly classification in surveillance videos

Multiple Instance Learning (MIL) is widely used to locate abnormal video frames in surveillance videos due to its ability to work with weakly-labeled data. On the other hand, graph-based semi-supervised approaches are employed to handle label sparsity-related issues such as Low Degree-of-Supervision (LDoS) and High Class-Imbalance (HCI). However, the application of the MIL paradigm in anomaly classification of surveillance videos faces significant challenges such as (i) LDoS, (ii) HCI, and (iii) A multitude of types of abnormal events and actions instead of a unique pattern. The current work proposes a novel anomaly classifier to address these issues with a new objective function (GssMILP) by leveraging the benefits from graph-based semi-supervised and multiple-instance learning approaches. In this regard, our contributions are six-fold. First, a deep hierarchical architecture (C3D+3DCNN BICLSTM) to extract generic video descriptors. These descriptors represent the multitude of types of abnormal events and actions by incorporating the local–global spatial information and long-short-term temporal information related to the object motion, human actions, and behavioral features. Second, we develop a learning objective (GssMILP) that extends the MIL-based instance-level labeler to graph-based semi-supervising. Third, we use three different regularizer terms, namely graph-based regularizer, non-convex L2-regularizer, and temporal-smoothness regularizer. Fourth, we present an elaborate experimental study comparing the performance of the proposed approach with six similar baseline approaches. The results indicate that at degree-of-supervision 1.0%, the proposed method improves the f1-score by 45.26%, 18.19%, 45.28%, and 9.22% in UCSD-ped1, UCSD-ped2, MED, and RWAD datasets, respectively. Fifth, we present an ablation study on the impact of different features on the performance of an anomaly classifier. Sixth, we provide temporal annotations for a label resource of 1900 videos in the RWAD dataset.

Scenario-Aware Recurrent Transformer for Goal-Directed Video Captioning

Fully mining visual cues to aid in content understanding is crucial for video captioning. However, most state-of-the-art video captioning methods are limited to generating captions purely based on straightforward information while ignoring the scenario and context information. To fill the gap, we propose a novel, simple but effective scenario-aware recurrent transformer (SART) model to execute video captioning. Our model contains a “scenario understanding” module to obtain a global perspective across multiple frames, providing a specific scenario to guarantee a goal-directed description. Moreover, for the sake of achieving narrative continuity in the generated paragraph, a unified recurrent transformer is adopted. To demonstrate the effectiveness of our proposed SART, we have conducted comprehensive experiments on various large-scale video description datasets, including ActivityNet, YouCookII, and VideoStory. Additionally, we extend a story-oriented evaluation framework for assessing the quality of the generated caption more precisely. The superior performance has shown that SART has a strong ability to generate correct, deliberative, and narrative coherent video descriptions.

Semantic context driven language descriptions of videos using deep neural network

The massive addition of data to the internet in text, images, and videos made computer vision-based tasks challenging in the big data domain. Recent exploration of video data and progress in visual information captioning has been an arduous task in computer vision. Visual captioning is attributable to integrating visual information with natural language descriptions. This paper proposes an encoder-decoder framework with a 2D-Convolutional Neural Network (CNN) model and layered Long Short Term Memory (LSTM) as the encoder and an LSTM model integrated with an attention mechanism working as the decoder with a hybrid loss function. Visual feature vectors extracted from the video frames using a 2D-CNN model capture spatial features. Specifically, the visual feature vectors are fed into the layered LSTM to capture the temporal information. The attention mechanism enables the decoder to perceive and focus on relevant objects and correlate the visual context and language content for producing semantically correct captions. The visual features and GloVe word embeddings are input into the decoder to generate natural semantic descriptions for the videos. The performance of the proposed framework is evaluated on the video captioning benchmark dataset Microsoft Video Description (MSVD) using various well-known evaluation metrics. The experimental findings indicate that the suggested framework outperforms state-of-the-art techniques. Compared to the state-of-the-art research methods, the proposed model significantly increased all measures, B@1, B@2, B@3, B@4, METEOR, and CIDEr, with the score of 78.4, 64.8, 54.2, and 43.7, 32.3, and 70.7, respectively. The progression in all scores indicates a more excellent grasp of the context of the inputs, which results in more accurate caption prediction.

Visual and language semantic hybrid enhancement and complementary for video description

Visual and language semantic hybrid enhancement and complementary for video description

Laparoscopic glissonean pedicle approach: step by step video description of the technique from different centres (with video).

Laparoscopic glissonean pedicle approach: step by step video description of the technique from different centres (with video).

DeepRide: Dashcam Video Description Dataset for Autonomous Vehicle Location-Aware Trip Description

Video description is one of the most challenging task in the combined domain of computer vision and natural language processing. Captions for various open and constrained domain videos have been generated in the recent past but descriptions for driving dashcam videos have never been explored to the best of our knowledge. With the aim to explore dashcam video description generation for autonomous driving, this study presents DeepRide: a large-scale dashcam driving video description dataset for location-aware dense video description generation. The human-described dataset comprises visual scenes and actions with diverse weather, people, objects, and geographical paradigms. It bridges the autonomous driving domain with video description by textual description generation of the visual information as seen by a dashcam. We describe 16,000 videos (40 seconds each) in English employing 2,700 man-hours by two highly qualified teams with domain knowledge. The descriptions consist of eight to ten sentences covering each dashcam video’s global features and event features in 60 to 90 words. The dataset consists of more than 130K sentences, totaling approximately one million words. We evaluate the dataset by employing location aware vision-language recurrent transformer framework to elaborate on the efficacy and significance of the visio-linguistics research for autonomous vehicles. We provided base line results to evaluate the dataset by employing three existing state-of-the-art recurrent models. The memory augmented transformer performed superior due to its highly summarized memory state for visual information and the sentence history while generating the trip description. Our proposed dataset opens a new dimension of diverse and exciting applications, such as self-driving vehicle reporting, driver and vehicle safety, inter-vehicle road intelligence sharing, and travel occurrence reports.

A Deep Learning-Based Approach for Inappropriate Content Detection and Classification of YouTube Videos

The exponential growth of videos on YouTube has attracted billions of viewers among which the majority belongs to a young demographic. Malicious uploaders also find this platform as an opportunity to spread upsetting visual content, such as using animated cartoon videos to share inappropriate content with children. Therefore, an automatic real-time video content filtering mechanism is highly suggested to be integrated into social media platforms. In this study, a novel deep learning-based architecture is proposed for the detection and classification of inappropriate content in videos. For this, the proposed framework employs an ImageNet pre-trained convolutional neural network (CNN) model known as EfficientNet-B7 to extract video descriptors, which are then fed to bidirectional long short-term memory (BiLSTM) network to learn effective video representations and perform multiclass video classification. An attention mechanism is also integrated after BiLSTM to apply attention probability distribution in the network. These models are evaluated on a manually annotated dataset of 111,156 cartoon clips collected from YouTube videos. Experimental results demonstrated that EfficientNet-BiLSTM (accuracy = 95.66%) performs better than attention mechanism-based EfficientNet-BiLSTM (accuracy = 95.30%) framework. Secondly, the traditional machine learning classifiers perform relatively poor than deep learning classifiers. Overall, the architecture of EfficientNet and BiLSTM with 128 hidden units yielded state-of-the-art performance (f1 score = 0.9267). Furthermore, the performance comparison against existing state-of-the-art approaches verified that BiLSTM on top of CNN captures better contextual information of video descriptors in network architecture, and hence achieved better results in child inappropriate video content detection and classification.

AAP-MIT: Attentive Atrous Pyramid Network and Memory Incorporated Transformer for Multisentence Video Description.

Generating multi-sentence descriptions for video is considered to be the most complex task in computer vision and natural language understanding due to the intricate nature of video-text data. With the recent advances in deep learning approaches, the multi-sentence video description has achieved an impressive progress. However, learning rich temporal context representation of visual sequences and modelling long-term dependencies of natural language descriptions is still a challenging problem. Towards this goal, we propose an Attentive Atrous Pyramid network and Memory Incorporated Transformer (AAP-MIT) for multi-sentence video description. The proposed AAP-MIT incorporates the effective representation of visual scene by distilling the most informative and discriminative spatio-temporal features of video data at multiple granularities and further generates the highly summarized descriptions. Profoundly, we construct AAP-MIT with three major components: i) a temporal pyramid network, which builds the temporal feature hierarchy at multiple scales by convolving the local features at temporal space, ii) a temporal correlation attention to learn the relations among various temporal video segments, and iii) the memory incorporated transformer, which augments the new memory block in language transformer to generate highly descriptive natural language sentences. Finally, the extensive experiments on ActivityNet Captions and YouCookII datasets demonstrate the substantial superiority of AAP-MIT over the existing approaches.

Explanations of steam as liberal arts by cooking of steamed custard pudding

The video explaining how to cook steamed custard pudding also explains steam saturated vapor pressure. The video is streamed by YouTube channel of Oita University, and the content of the video should be understand by common Japanese. Although the bubble in a custard pudding may not be fail in the world, the bubble in a custard pudding is fail in Japan, and the bubble is called “su”. The video explains how to suppress the bubble generation. Cooking method of caramel source is not explained in the video. The cooking method of caramel source is explained in the video description of the web page in YouTube. The video introduces 4 points for cooking steamed custard pudding without too much bubble which is called “su” by Japanese. The four points are , avoiding steam condensation on pudding surface, pressure release of steamer, temperature estimation by observing steam behavior, and confirming pudding coagulation by observation.

Multi-level Video Captioning based on Label Classification using Machine Learning Techniques

Video captioning is the heuristic and most essential task in the current world to save time by converting long and highly content-rich videos into simple and readable reports in text form. It is narrating the events happening in videos in natural language sentences. It makes the way to many more interesting tasks by the use of labels, tags, and terms such as video content retrieval, video search, video tagging, etc. Video captioning is currently being attempted by many researchers using some exciting Deep learning techniques. But this approach is to find the best of machine learning for the process of captioning videos in a different way. The novel part of the proposed approach is classifying videos by using the labels existing in video frames that belong to the various categories and producing consecutive Multi-Level captions that describe the entire video in a round-robin way. Informative features are extracted from the video frames such as Gray Level Co-occurrence Matrix (GLCM) features, Hu moments, and Statistical features to provide optimal results. This model is designed with two superior and optimal classifiers such as Support Vector Machine (SVM) and Naive Bayes separately. The models are demonstrated with the prevailing standard dataset Microsoft Research Video Description corpus (MSVD) and evaluated by the benchmark classification metrics such as Accuracy, Precision, Recall, and F1-Score.

CINEMATOGRAPHY AS A DESCRIPTIVE PHILOSOPHY "ON THE CINEFILM" OF IMAGES OF HOMO: FROM THE FIXATION OF IMAGES (HOMO PHOTOGRAPHICUS) TO THE TREACHERY OF IMAGES (CONTEMPORARY HOMO VIDENS)

The article explores art of cinematography as an objectified cultural reality, in spatial and temporal structures of video description. Genesis of art of photography has changed the habits of human perception and thinking process – from photographic images of Homo photographicus to contemporary Homo Videns whose visual experience is the experience of appropriation and distancing from the world in its optical modality. The art of cinematography, correlating with metaphysical discourses of reality, creates a socio‐cultural code as a matrix of meanings and senses. It provides organization and support to the semiotic fund of culture through performing the functions of socialization and cultural integration. The differences between video description and philosophical text are emphasized. The advantage of video description is being devoid of the complexity of philosophical texts due to readability of indexes, demonstration of codes that define their sequence through the captured images‐characters close to everyday life to the extent that they do not require any analysis as being self‐obvious. The magic function of screen image is that it has multiple reflections, so narrative of identity is transferred to the outer plane, onto the other side of the screen when the image created by the operator is a signal that triggers imagination. The indicative character is transferred to the image, and these indicators are indexed sufficiently to be stimulus signs, effects for Homo Videns’s perception. They are deciphered according to the standards and ideals of a particular culture whose codes are recognized by the subject of that culture at a particular moment which influences constitution and re‐constitution of Homo Videns who consumes both things and images – but also transformation of culture itself. Technological transformation of images of Homo Videns is due to changes in the technology of filming and cinematography which is an important segment of cultural history. Even if cameraman's skills do not directly determine movement of technical progress, they dwell on the achievements of technological revolutions. The choice of problematic fields of human images analysis as bound with evolution of rendering techniques and visualizations, today is more justified than the rationalization of these images.

Video conferencing software selection based on hybrid MCDM and cumulative prospect theory under a major epidemic

The video conferencing software is regarded as a significant tool for social distancing and getting incorporations up and going. Due to the indeterminacy of epidemic evolution and the multiple criteria, this paper proposes a video conferencing software selection method based on hybrid multi-criteria decision making (HMCDM) under risk and cumulative prospect theory (CPT), in which the criteria values are expressed in various mathematical forms (e.g., real numbers, interval numbers, and linguistic terms) and can be changed with natural states of the epidemic. Initially, the detailed description of video conferencing software selection problem under an epidemic are given. Subsequently, a whole procedure for video conferencing software selection is conducted, the approaches for processing and normalizing the multi-format evaluation values are presented. Furthermore, the expectations provided by DMs under different natural states of the epidemic are considered as the corresponding reference points (RP). Based on this, the matrix of gains and losses is constructed. Then, the prospect values of all criteria and the perceived probabilities of natural states are calculated according to the value function and the weighting function in CPT respectively. Finally, the proposed method is illustrated by an empirical case study, and the comparison analysis and the sensitivity analysis for the loss aversion parameter are conducted to prove the effectiveness and robustness. The results show that considering the psychological characteristics of DMs in selection decision is beneficial to avoid the unacceptable and potential loss risks. This study could provide a useful guideline for managers who intend to select appropriate video conferencing software.

Research on Human Motion Analysis in Moving Scene Based on Timing Detection and Video Description Algorithm

Technical movement analysis requires specialized domain knowledge and processing a large amount of data, and the advantages of AI in processing data can improve the efficiency of data analysis. In this paper, we propose a feature pyramid network-based temporal action detection (FPN-TAD) algorithm, which is used to solve the problem that the action proposal module has a low recall rate for small-scale temporal target action regions in the current video temporal action detection algorithm research. This paper is divided into three parts. The first part is an overview of the algorithm; the second part elaborates the network structure and the working principle of the FPN-TAD algorithm; and the third part gives the experimental results and analysis of the algorithm.

Looking "Cherry Red Spot Myoclonus" in the Eyes: Clinical Phenotype, Treatment Response, and Eye Movements in Sialidosis Type 1.

Sialidosis type 1 is a rare lysosomal storage disorder caused by mutations of the neuraminidase gene. Specific features suggesting this condition include myoclonus, ataxia and macular cherry-red spots. However, phenotypic variability exists. Here, we present detailed clinical and video description of three patients with this rare condition. We also provide an in-depth characterization of eye movement abnormalities, as an additional tool to investigate pathophysiological mechanisms and to facilitate diagnosis. In our patients, despite phenotypic differences, eye movement deficits largely localized to the cerebellum.

Content of YouTube videos on cassava production and processing in Nigeria


 
 
 This study examined content of YouTube videos on cassava production and processing posted in Nigeria between 2009 and 2019. Purposive sampling method was used in selecting 155 YouTube videos with cassava production and processing contents. Primary data on video source, content, duration, quality, number of views, subscribers, likes, comments, presentation format and year of upload were obtained by watching these videos. Data were analysed using frequency counts, percentages, means, standard deviation. Many (53.5%) of the videos were on processing of cassava, while most (80.0%) of the videos were relatively recent (2014 - 2019). Private individuals (38.7%) and media houses (34.2%) were the main sources of videos on cassava production and processing on YouTube. The major content of the video was on agronomic practices in cassava production (12.9%) and value addition (11.0%). Also, 66.0% of the videos had video description (descriptive texts), 36.8% had mobilizing information while 52.3% had a video quality of 720p. Furthermore, 72.9% of the videos had between 1-to-7-minute runtime and these videos had more likes than dislikes with mean values of 92.8 and 5.6 respectively. The mean number of views, subscribers and comments were 11,138.1, 179,537.6 and 13.6, respectively. The presentation patterns in the videos were in form of news (24.5%) and documentary (20.0%). YouTube videos on cassava production and processing were well viewed with a substantial number of subscribers. More YouTube videos on cassava production and processing should be produced with varied contents by different stakeholders in the agricultural sector.
 
 

Predicting Popularity of Video Streaming Services with Representation Learning: A Survey and a Real-World Case Study.

The Internet’s popularization has increased the amount of content produced and consumed on the web. To take advantage of this new market, major content producers such as Netflix and Amazon Prime have emerged, focusing on video streaming services. However, despite the large number and diversity of videos made available by these content providers, few of them attract the attention of most users. For example, in the data explored in this article, only 6% of the most popular videos account for 85% of total views. Finding out in advance which videos will be popular is not trivial, especially given many influencing variables. Nevertheless, a tool with this ability would be of great value to help dimension network infrastructure and properly recommend new content to users. In this way, this manuscript examines the machine learning-based approaches that have been proposed to solve the prediction of web content popularity. To this end, we first survey the literature and elaborate a taxonomy that classifies models according to predictive features and describes state-of-the-art features and techniques used to solve this task. While analyzing previous works, we saw an opportunity to use textual features for video prediction. Thus, additionally, we propose a case study that combines features acquired through attribute engineering and word embedding to predict the popularity of a video. The first approach is based on predictive attributes defined by resource engineering. The second takes advantage of word embeddings from video descriptions and titles. We experimented with the proposed techniques in a set of videos from GloboPlay, the largest provider of video streaming services in Latin America. A combination of engineering features and embeddings using the Random Forest algorithm achieved the best result, with an accuracy of 87%.

Dynamic texture representation based on oriented magnitudes of Gaussian gradients

Efficiently capturing shape and turbulent motions of dynamic textures (DTs) for video description is a challenge in real applications due to the negative influences of the well-known problems: environmental elements, illumination, scale, and noise. In this paper, we propose an efficient and simple framework for DT representation based on the oriented features of high-order Gaussian gradients. Firstly, 2D/3D Gaussian-based filtering kernels in high-order partial derivatives are taken into account the video analysis as a preprocessing step to obtain corresponding gradient-filtered images/volumes. After that, the oriented features, which are robust against the above issues, are extracted by decomposing the Gaussian derivative magnitudes into oriented components. Finally, a shallow local encoding is utilized for structuring spatio-temporal features from these oriented magnitudes. This allows constructing discriminative descriptors with promising performances compared to those based on the non-oriented ones. Experimental results for DT classification task on benchmark datasets have verified the interest of our proposal.

Robotic Assisted Laparoscopic Tubal Anastomosis

<h3>Study Objective</h3> To demonstrate a technique for tubal anastomosis. <h3>Design</h3> A video description of steps of a technique for minimally invasive tubal anastomosis. <h3>Setting</h3> video description of surgical technique using robotic assistance. <h3>Patients or Participants</h3> A 33-years old female with past history of sterilization with partial salpingectomy and no other infertility factors who decided to regain her fertility potential, underwent a laparoscopic tubal anastomosis. <h3>Interventions</h3> Laparoscopic tubal anastomosis with robotic assistance. <h3>Measurements and Main Results</h3> Description of surgical technique using surgical video. <h3>Conclusion</h3> tubal anastomosis using robotic assistance is a reproducible minimally invasive approach.

Tips and Tricks for Deep Endometriosis in the Frozen Pelvis

Study Objective The objective of this study was to describe an approach to deep endometriosis in the frozen pelvis. Given it's high complexity, these cases must be thoroughly studied by every pelvic surgeon. Design Video description of tips and tricks to navigate through the frozen pelvis. Setting N/A. Patients or Participants The surgery was performed in a 20 years oldwoman who suffered from extensive retrocervical endometriotic lesions with obliteration of the pouch of Douglas. She presented with dysmenorrhea, dyspareunia, dyschezia and tenesmus. Her diagnosis was made through physical examination, ultrasound and MRI. She had an endometrioma in her left ovary and a lesion on the intestinal wall with 1.6 cm of infiltration and 3.5cm of extension, totaling 32% of the intestinal loop diameter. Interventions We start the surgery through the development of the avascular anatomical spaces. To loosen the intestinal lesion of the uterine wall, we traction the rectum centrally and posteriorly, and dissect towards the rectovaginal septum, passing the lesion area. In cases of endometriosis with extension to the ovarian pit, we like to remove the posterior leaflet of the broad ligament in conjunction with the uterosacral ligament while the assistant surgeon keeps the ureter and hypogastric nerve apart. Finally, after circumventing all the lesion and preserving the pelvic innervation, the excess fibrotic process must be resected by shaving the intestinal wall to reduce the lesion size and allow the discoid resection. Measurements and Main Results We conclude that the surgical technique in complex frozen pelvis cases must be flawless. Usually, the anatomy is very distorted and is more demanding to the surgeon. Thus, the correct use of energy devices and the help of the surgical assistant is key for successful outcome. Conclusion In summary, describe useful tips and tricks that equip the surgeon to navigate through the complex anatomy of a frozen pelvis. The objective of this study was to describe an approach to deep endometriosis in the frozen pelvis. Given it's high complexity, these cases must be thoroughly studied by every pelvic surgeon. Video description of tips and tricks to navigate through the frozen pelvis. N/A. The surgery was performed in a 20 years oldwoman who suffered from extensive retrocervical endometriotic lesions with obliteration of the pouch of Douglas. She presented with dysmenorrhea, dyspareunia, dyschezia and tenesmus. Her diagnosis was made through physical examination, ultrasound and MRI. She had an endometrioma in her left ovary and a lesion on the intestinal wall with 1.6 cm of infiltration and 3.5cm of extension, totaling 32% of the intestinal loop diameter. We start the surgery through the development of the avascular anatomical spaces. To loosen the intestinal lesion of the uterine wall, we traction the rectum centrally and posteriorly, and dissect towards the rectovaginal septum, passing the lesion area. In cases of endometriosis with extension to the ovarian pit, we like to remove the posterior leaflet of the broad ligament in conjunction with the uterosacral ligament while the assistant surgeon keeps the ureter and hypogastric nerve apart. Finally, after circumventing all the lesion and preserving the pelvic innervation, the excess fibrotic process must be resected by shaving the intestinal wall to reduce the lesion size and allow the discoid resection. We conclude that the surgical technique in complex frozen pelvis cases must be flawless. Usually, the anatomy is very distorted and is more demanding to the surgeon. Thus, the correct use of energy devices and the help of the surgical assistant is key for successful outcome. In summary, describe useful tips and tricks that equip the surgeon to navigate through the complex anatomy of a frozen pelvis.