Video Data Research Articles

Fish community structure and diversity partitioning in the nearshore benthoscapes of San Salvador, Bahamas

Exploring variation in community structure and species diversity across spatial scales provides insights into factors structuring communities and promoting diversity and informs management and conservation. This study focused on spatial variation in fish community structure and diversity in the nearshore benthoscapes of San Salvador, Bahamas. Data were collected at thirteen sites across five regions and three different habitat types (patch reefs, high relief, and rubble sites) using video and roving snorkeler observations, which were analyzed separately. Although there were similar sets of dominant species around the island, multivariate analyses indicated significant differences in community structure among regions and sites and at the within-site scale, which can be attributed to differences among habitat types and localized environmental effects at the site level. A total of 55 species were found in the video data; 141 species were found in the snorkeler observations, likely due to the presence of a greater variety of habitats (deeper patch reefs, seagrass beds and mangroves), larger area observed, and increased likelihood to observe cryptic species. Diversity partitioning indicated that β₂ (among sites within a region) and β₃ (among different regions) diversity contributed greater than expected numbers of species to overall γ-diversity. The different regions around the island and the mix of habitats within each region may provide some measure of resilience in fish communities to disturbances and changing environmental conditions. Our results underscore the importance of protecting mixes of habitat in conservation and management efforts focused on the nearshore benthoscapes of tropical islands such as San Salvador.

Learning to Answer Visual Questions from Web Videos.

Recent methods for visual question answering rely on large-scale annotated datasets. Manual annotation of questions and answers for videos, however, is tedious, expensive and prevents scalability. In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision. We leverage a question generation transformer trained on text data and use it to generate question-answer pairs from transcribed video narrations. Given narrated videos, we then automatically generate the HowToVQA69M dataset with 69M video-question-answer triplets. To handle the open vocabulary of diverse answers in this dataset, we propose a training procedure based on a contrastive loss between a video-question multi-modal transformer and an answer transformer. We introduce the zero-shot VideoQA task and the VideoQA feature probe evaluation setting and show excellent results. Furthermore, our method achieves competitive results on MSRVTT-QA, ActivityNet-QA, MSVD-QA and How2QA datasets. We also show that our approach generalizes to another source of web video and text data. We generate the WebVidVQA3M dataset from videos with alt-text annotations, and show its benefits for training VideoQA models. Finally, for a detailed evaluation we introduce iVQA, a new VideoQA dataset with reduced language bias and high-quality manual annotations.

Comparison of traditional and molecular surveys of fish biodiversity in southern Te Wāhipounamu/Fiordland (Aotearoa/New Zealand)

AbstractEffective management of biodiversity requires regular surveillance of multiple species. Analysis of environmental DNA (eDNA) by metabarcoding holds promise to achieve this relatively easily. However, taxonomy‐focused eDNA surveys need suitable molecular reference data, which are often lacking, particularly at the species level and for remote locations. To evaluate the comparability of environmental DNA surveys and traditional surveys in a real‐life case study in a marine area of high conservation value, we conducted a biodiversity survey of the fish in remote and pristine Te Wāhipounamu/Fiordland (Aotearoa/New Zealand), incorporating multiple data sources. We compared eDNA‐derived species identifications against Baited Remote Underwater Video (BRUV) data collected at the same time and locations as eDNA. We also cross‐referenced both eDNA and BRUV data against literature and the Ocean Biodiversity Information System (OBIS), with literature and OBIS data representing a summary of multiple traditional surveying approaches. In total, we found 116 fish species in our study area. Environmental DNA detected 43 species; however, only three of those species overlap with species known from the literature, OBIS, or our BRUV analyses. A total of 61 fish species were known from the region from the literature, while OBIS listed 28 species, and our BRUV analyses picked up 26 species. BRUV data coincided more strongly than eDNA data with literature and OBIS data. Twenty of the 26 species detected by BRUV were known from literature and OBIS. We argue that limitated DNA reference databases are the main cause of this discrepancy, and our results indicate that eDNA of rare and endangered species can be detected if matching reference data were available. Environmental DNA analyses can only identify species present among reference data and with relaxed taxonomic assignment parameters may converge on relatives of detected species if the actually existing species themselves are missing among reference data. However, the high number of species detected by our eDNA analyses confirms that eDNA could be a powerful tool for biodiversity surveys if suitable investments in local reference databases were made.

Study on the Application of Improved Deep Convolutional Neural Network Algorithm in Broken Information Recovery

Abstract This study on the fusion of deep convolutional neural network (CNN) and extended short-term memory network (LSTM) aims to improve the efficiency and accuracy of broken information recovery. The challenges faced by traditional information recovery techniques are addressed through improved algorithms. The research methodology includes constructing CNN models to automatically extract features and combining LSTM networks to process complex time-series data. We conducted a detailed experimental evaluation of the CNN-LSTM fusion algorithm, including recovery of different types of corrupted data, and compared it with other algorithms. The results show that the CNN-LSTM fusion algorithm has the highest structural similarity (0.9545) and the most minor normalized mean square error (0.0016) for recovering broken video information, outperforming the methods using CNN or LSTM alone. The fusion algorithm dramatically reduces computation time and resource consumption for processing complex datasets. The combination of CNN and LSTM significantly improves the performance of broken information recovery, especially in processing video and audio data, and provides new ideas for future information processing techniques.

ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation.

Natural Language Generation (NLG) accepts input data in the form of images, videos, or text and generates corresponding natural language text as output. Existing NLG methods mainly adopt a supervised approach and rely heavily on coupled data-to-text pairs. However, for many targeted scenarios and for non-English languages, sufficient quantities of labeled data are often not available. As a result, it is necessary to collect and label data-text pairs for training, which is both costly and time-consuming. To relax the dependency on labeled data of downstream tasks, we propose an intuitive and effective zero-shot learning framework, ZeroNLG, which can deal with multiple NLG tasks, including image-to-text (image captioning), video-to-text (video captioning), and text-to-text (neural machine translation), across English, Chinese, German, and French within a unified framework. ZeroNLG does not require any labeled downstream pairs for training. During training, ZeroNLG (i) projects different domains (across modalities and languages) to corresponding coordinates in a shared common latent space; (ii) bridges different domains by aligning their corresponding coordinates in this space; and (iii) builds an unsupervised multilingual auto-encoder to learn to generate text by reconstructing the input text given its coordinate in shared latent space. Consequently, during inference, based on the data-to-text pipeline, ZeroNLG can generate target sentences across different languages given the coordinate of input data in the common space. Within this unified framework, given visual (imaging or video) data as input, ZeroNLG can perform zero-shot visual captioning; given textual sentences as input, ZeroNLG can perform zero-shot machine translation. We present the results of extensive experiments on twelve NLG tasks, showing that, without using any labeled downstream pairs for training, ZeroNLG generates high-quality and "believable" outputs and significantly outperforms existing zero-shot methods. Our code and data are available at https://github.com/yangbang18/ZeroNLG.

MCD-Net: Toward RGB-D Video Inpainting in Real-World Scenes.

Video inpainting gains an increasing amount of attention ascribed to its wide applications in intelligent video editing. However, despite tremendous progress made in RGB video inpainting, the existing RGB-D video inpainting models are still incompetent to inpaint real-world RGB-D videos, as they simply fuse color and depth via explicit feature concatenation, neglecting the natural modality gap. Moreover, current RGB-D video inpainting datasets are synthesized with homogeneous and delusive RGB-D data, which is far from real-world application and cannot provide comprehensive evaluation. To alleviate these problems and achieve real-world RGB-D video inpainting, on one hand, we propose a Mutually-guided Color and Depth Inpainting Network (MCD-Net), where color and depth are reciprocally leveraged to inpaint each other implicitly, mitigating the modality gap and fully exploiting cross-modal association for inpainting. On the other hand, we build a Video Inpainting with Depth (VID) dataset to supply diverse and authentic RGB-D video data with various object annotation masks to enable comprehensive evaluation for RGB-D video inpainting under real-world scenes. Experimental results on the DynaFill benchmark and our collected VID dataset demonstrate our MCD-Net not only yields the state-of-the-art quantitative performance but successfully achieves high-quality RGB-D video inpainting under real-world scenes. All resources are available at https://github.com/JCATCV/MCD-Net.

Concept Graph Neural Networks for Surgical Video Understanding.

Analysis of relations between objects and comprehension of abstract concepts in the surgical video is important in AI-augmented surgery. However, building models that integrate our knowledge and understanding of surgery remains a challenging endeavor. In this paper, we propose a novel way to integrate conceptual knowledge into temporal analysis tasks using temporal concept graph networks. In the proposed networks, a knowledge graph is incorporated into the temporal video analysis of surgical notions, learning the meaning of concepts and relations as they apply to the data. We demonstrate results in surgical video data for tasks such as verification of the critical view of safety, estimation of the Parkland grading scale as well as recognizing instrument-action-tissue triplets. The results show that our method improves the recognition and detection of complex benchmarks as well as enables other analytic applications of interest.

Discourse in Research-Practice Partnership Meetings: A Comparison of Conditions Across Contexts

Research-practice partnerships (RPPs) are gaining international attention as they promise to close the gap between research and practice in education. As RPPs bring together participants with diverse expertise, how people dialogue to collectively learn and address problems of practice is critically important. Analyzing video data from RPP meetings in Switzerland and the United States, this micro-analytic study examines the extent of generative discourse in RPP meetings and the conditions under which it occurs. Since RPPs vary significantly along a number of dimensions, it is useful to compare them to see how these features influence discourse and learning. Across the Swiss and US contexts, almost half of the meeting time was generative—altering meanings and/or creating new knowledge and perspectives. Discourse patterns varied, however, reflecting the different sizes, purposes, and activity structures of the RPPs. Examining meeting discourse across conditions may help promote knowledge generation and continuous improvement in RPPs.

Classroom Supports for Generalizing

Generalizing is a critical aspect of mathematics learning, with researchers and policy documents highlighting generalizing as a core mathematical practice. It can also be challenging to foster in class settings, and teachers need access to better resources to teach generalizing, including an understanding of effective forms of instruction. This article proposes Classroom Supports for Generalizing (CSGs), investigating how multiple elements—such as tasks, teacher moves, student interactions, and representations—interact to meaningfully foster student generalizing. Drawing on class video data from a middle school teacher and two high school teachers, we present the CSG Framework, which identifies three categories of supports: Interactions for Generalizing, Structures for Generalizing, and Routines for Generalizing.

A preliminary analysis for determining passenger car equivalent value for motorcycle during first lost time and saturation time at pretimed signalized intersections

In Banda Aceh City, signalized intersections are a strategic approach to mitigate conflicts such as accidents and traffic congestion. This was achieved by efficiently managing the flow of vehicles passing through intersections, enhancing their overall capacity. Currently, the management of signalized intersections in Banda Aceh still adheres to the guidelines stipulated in the 1997 Indonesian Highway Capacity Manual (MKJI 1997). Given the considerable variation in driver behavior and vehicle composition, particularly motorcycles, within Banda Aceh, a need for updated practices arises. Therefore, this research determined the comparison of PCEs values at the initial and saturation time based on the PKJI 2023 and the new calculations. It was conducted at four-legged intersections, namely Pocut Baren, AMD, Lhong Raya, and Jambo Tape. The primary dataset comprised intersections geometry, effective green time, traffic flow composition captured through drone recordings, and signal configurations. Statistical analysis in this research used multiple linear regression analysis to estimate the Passenger Car Unit (PCU) model parameters using video data recorded at the four-legged intersections. The result showed that PCE values for motorcycles during the initial time was 0.16, higher than the average saturation time values of 0.08.

A novel approach for enhanced abnormal action recognition via coarse and precise detection stage

<abstract> <p>With the proliferation of urban video surveillance systems, the abundance of surveillance video data has emerged as a pivotal asset for enhancing public safety. Within these video archives, the identification of abnormal human actions carries profound implications for security incidents. Nevertheless, existing surveillance systems primarily rely on conventional algorithms, leading to both missed incidents and false alarms. To address the challenge of automating multi-object surveillance video analysis, this study introduces a comprehensive method for the detection and recognition of multi-object abnormal actions. This study comprises a two-stage framework: the coarse detection stage employs an enhanced YOWOv2E model for spatio-temporal action detection, while the precise detection stage utilizes a two-stream network for precise action classification. In parallel, this paper presents the PSA-Dataset to address the current limitations in the field of abnormal action detection. Experimental results, collected from both public datasets and a self-built dataset, illustrate the effectiveness of the proposed method in identifying a wide spectrum of abnormal actions. This work offers valuable insights for automating the analysis of human actions in videos pertaining to public security.</p> </abstract>

Cube-Evo: A Query-Efficient Black-Box Attack on Video Classification System

The current progressive research in the domain of black-box adversarial attack enhances the reliability of deep neural network (DNN)-based video systems. Recent works mainly carry out black-box adversarial attacks on video systems by query-based parameter dimension reduction. However, the additional temporal dimension of video data leads to massive query consumption and low attack success rate. In this article, we embark on our efforts to design an effective adversarial attack on popular video classification systems. We deeply root the observations that the DNN-based systems are sensitive to adversarial perturbations with high frequency and reconstructed shape. Specifically, we propose a systematic attack pipeline Cube-Evo, aiming to reduce the search space dimension and obtain the effective adversarial perturbation via the optimal parameter group updating. We evaluate the proposed attack pipeline on two popular datasets: UCF101 and JESTER. Our attack pipeline reduces query consumption and achieves a high success rate on various DNN-based video classification systems. Compared with the state-of-the-art method Geo-Trap-Att, our pipeline averagely reduces 1.6× query consumption in untargeted attacks and 2.9× in targeted attacks. Besides, Cube-Evo improves 13% attack success rate on average, achieving new state-of-the-art results over diverse video classification systems.

K-means clustering based analysis for video data and their high acceptance in universities students’ science popularization education

K-means clustering based analysis for video data and their high acceptance in universities students’ science popularization education

Skeleton action recognition via graph convolutional network with self-attention module

<abstract><p>Skeleton-based action recognition is an important but challenging task in the study of video understanding and human-computer interaction. However, existing methods suffer from two deficiencies. On the one hand, most methods usually involve manually designed convolution kernel which cannot capture spatial-temporal joint dependencies of complex regions. On the other hand, some methods just use the self-attention mechanism, ignoring its theoretical explanation. In this paper, we proposed a unified spatio-temporal graph convolutional network with a self-attention mechanism (SA-GCN) for low-quality motion video data with fixed viewing angle. SA-GCN can extract features efficiently by learning weights between joint points of different scales. Specifically, the proposed self-attention mechanism is end-to-end with mapping strategy for different nodes, which not only characterizes the multi-scale dependencies of joints, but also integrates the structural features of the graph and an ability of self-learning fusion features. Moreover, the attention mechanism proposed in this paper can be theoretically explained by GCN to some extent, which is usually not considered in most existing models. Extensive experiments on two widely used datasets, NTU-60 RGB+D and NTU-120 RGB+D, demonstrated that SA-GCN significantly outperforms a series of existing mainstream approaches in terms of accuracy.</p></abstract>

Coverage estimation of benthic habitat features by semantic segmentation of underwater imagery from South-eastern Baltic reefs using deep learning models

Underwater imagery (UI) is an important and sometimes the only tool for mapping hard-bottom habitats. With the development of new camera systems, from hand-held or simple “drop-down” cameras to ROV/AUV-mounted video systems, video data collection has increased considerably. However, the processing and analysing of vast amounts of imagery can become very labour-intensive, thus making it ineffective both time-wise and financially. This task could be simplified if the processes or their intermediate steps could be done automatically. Luckily, the rise of AI applications for automatic image analysis tasks in the last decade has empowered researchers with robust and effective tools. In this study, two ways to make UI analysis more efficient were tested with eight dominant visual features of the Southeastern Baltic reefs: 1) the simplification of video processing and expert annotation efforts by skipping the video mosaicking step and reducing the number of frames analysed; 2) the application of semantic segmentation of UI using deep learning models. The results showed that the annotation of individual frames provides similar results compared to 2D mosaics; moreover, the reduction of frames by 2–3 times resulted in only minor differences from the baseline. Semantic segmentation using the PSPNet model as the deep learning architecture was extensively evaluated, applying three variants of validation. The accuracy of segmentation, as measured by the intersection-over-union, was mediocre; however, estimates of visual coverage percentages were fair: the difference between the expert annotations and model-predicted segmentation was less than 6–8%, which could be considered an encouraging result.

Speech recognition enhancement based on wireless network sensors application in interactive intelligent teaching system

The speech function used in the multi-dimensional speech recognition model based on MTL and RNN is the commonly used MFCC function, and in feature extraction, after various filtering and conversion, deleting part of the speech information is a requirement for 3D audio recognition. Use multidimensional information as much as possible. Based on this, this article not only brings the development of convolutional neural network intelligent teaching, but also proposes a new education model. Heuristic education like flipped classroom is still the content of education research in recent years, and it is important to the current education form in China. It is mainly based on the traditional lecture mode or the classroom form. This article focuses on undergraduate education. It is designed in the classroom based on the video data of the undergraduate interactive activities and the interactive system. It has important research significance and practicality. At the same time, the interactive content of the traditional teaching method is used to perform voice recognition, and the recognition effect of the main body of the interactive system is further improved through the voice information. The spatial perspective is used to divide the lens perspective into seminars and experimental forms, and the pixel change value of each frame provides a specific reference for it to determine the reflection of student interaction activities and the evaluation of student seminars. This article introduces the implementation process of interactive content and interactive activities of the teaching system in the classroom, introduces the speech recognition framework of the MARS Chinese teaching system, uses the model principle, and compares the general speech data set test with the actual ARST model audio. The results show that, MAS model performance is more advantageous.

Contact-Free Atrial Fibrillation Screening with Attention Network.

Atrial Fibrillation (AF) screening from face videos has become popular with the trend of telemedicine and telehealth in recent years. In this study, the largest facial image database for camera-based AF detection is proposed. There are 657 participants from two clinical sites and each of them is recorded for about 10 minutes of video data, which can be further processed as over 10,000 segments around 30 seconds, where the duration setting is referred to the guideline of AF diagnosis. It is also worth noting that, 2,979 segments are segment-wise labeled, that is, every rhythm is independently labeled with AF or not. Besides, all labels are confirmed by the cardiologist manually. Various environments, talking, facial expressions, and head movements are involved in data collection, which meets the situations in practical usage. Specific to camera-based AF screening, a novel CNN-based architecture equipped with an attention mechanism is proposed. It is capable of fusing heartbeat consistency, heart rate variability derived from remote photoplethysmography, and motion features simultaneously to reliable outputs. With the proposed model, the performance of intra-database evaluation comes up to 96.62% of sensitivity, 90.61% of specificity, and 0.96 of AUC. Furthermore, to check the capability of adaptation of the proposed method thoroughly, the cross-database evaluation is also conducted, and the performance also reaches about 90% on average with the AUCs being over 0.94 in both clinical sites.

Informatization Exploration of Wushu Teaching Management Platform in Colleges and Universities under the Concept of Modern Education

Abstract In the evolving landscape of higher education, integrating information technology into Wushu teaching management presents an opportunity to elevate teaching standards. This paper investigates the development of an innovative platform for Wushu instruction, utilizing cutting-edge video analysis techniques to enhance learning outcomes. By analyzing video data from Hikvision DS-2CD2051-1 webcams with advanced preprocessing methods, we demonstrate the effectiveness of the Single Shot MultiBox Detector (SSD) algorithm in improving action recognition over conventional CNN approaches. Our results reveal a notable enhancement in the quality of Wushu movement evaluation, evidenced by a 72.02% accuracy rate and a 0.3865 Mean Squared Error (MSE). The deployment of this technology in a university setting not only fosters a significant uplift in student performance but also boosts their engagement and satisfaction levels. This paper contributes to the discourse on educational technology by offering a successful model for the digital Transformation of Wushu teaching in academic institutions.

Hypergraph-Based Multi-View Action Recognition Using Event Cameras.

Action recognition from video data forms a cornerstone with wide-ranging applications. Single-view action recognition faces limitations due to its reliance on a single viewpoint. In contrast, multi-view approaches capture complementary information from various viewpoints for improved accuracy. Recently, event cameras have emerged as innovative bio-inspired sensors, leading to advancements in event-based action recognition. However, existing works predominantly focus on single-view scenarios, leaving a gap in multi-view event data exploitation, particularly in challenges like information deficit and semantic misalignment. To bridge this gap, we introduce HyperMV, multi-view event-based action recognition framework. HyperMV converts discrete event data into frame-like representations and extracts view-related features using a shared convolutional network. By treating segments as vertices and constructing hyperedges using rule-based and KNN-based strategies, a multi-view hypergraph neural network that captures relationships across viewpoint and temporal features is established. The vertex attention hypergraph propagation is also introduced for enhanced feature fusion. To prompt research in this area, we present the largest multi-view event-based action dataset THUMV-EACT-50, comprising 50 actions from 6 viewpoints, which surpasses existing datasets by over tenfold. Experimental results show that HyperMV significantly outperforms baselines in both cross-subject and cross-view scenarios, and also exceeds the state-of-the-arts in frame-based multi-view action recognition.

Exploring Score-Level and Decision-Level Fusion of Inertial and Video Data for Intake Gesture Detection

Exploring Score-Level and Decision-Level Fusion of Inertial and Video Data for Intake Gesture Detection