Video Dataset Research Articles

Real-Time Decoding of Snapshot Compressive Imaging Using Tensor FISTA-Net.

Snapshot compressive imaging (SCI) cameras compress high-speed videos or hyperspectral images into measurement frames. However, decoding the data frames from measurement frames is compute-intensive. Existing state-of-the-art decoding algorithms suffer from low decoding quality or heavy running time or both, which are not practical for real-time applications. In this article, we exploit the powerful learning ability of deep neural networks (DNN) and propose a novel tensor fast iterative shrinkage-thresholding algorithm net (Tensor FISTA-Net) as a real-time decoder for SCI cameras. Since SCI cameras have an accurate physical model, we can trade training time for the decoding time by generating abundant synthetic data and training a decoder on the cloud. Tensor FISTA-Net not only learns a sparse representation of the frames through convolution layers but also reduces the decoding time and memory consumption significantly through tensor operations, which makes Tensor FISTA-Net an appropriate approach for a real-time decoder. Our proposed Tensor FISTA-Net obtains an average PSNR improvement of 0.79-2.84 dB (video images) and 2.61-4.43 dB (hyperspectral images) over the state-of-the-art algorithms, along with more clear and detailed visual results on real SCI datasets, Hammer and Wheel, respectively. Our Tensor FISTA-Net reaches 45 frames per second in video datasets and 70 frames per second in hyperspectral datasets, meeting the real-time requirement. Besides, the trained model occupies only a 12 -MB memory footprint, making it applicable to real-time Internet of Things (IoT) applications.

HOI-V: One-stage human-object interaction detection based on multi-feature fusion in videos

Effective detection of Human-Object Interaction (HOI) is important for machine understanding of real-world scenarios. Nowadays, image-based HOI detection has been abundantly investigated, and recent one-stage methods strike a balance between accuracy and efficiency. However, it is difficult to predict temporal-aware interaction actions from static images since limited temporal context information is introduced. Meanwhile, due to the lack of early large-scale video HOI datasets and the high computational cost of spatial-temporal HOI model training, recent exploratory studies mostly follow a two-stage paradigm, but independent object detection and interaction recognition still suffer from computational redundancy and independent optimization. Therefore, inspired by the one-stage interaction point detection framework, a one-stage spatial-temporal HOI detection baseline is proposed in this paper, in which the short-term local motion features and long-term temporal context features are obtained by the proposed temporal differential excitation module (TDEM) and DLA-TSM backbone. Complementary visual features between multiple clips are then extracted by multi-feature fusion and fed into the parallel detection branches. Finally, a video dataset containing only actions with reduced data size (HOI-V) is constructed to motivate further research on end-to-end video HOI detection. Extensive experiments are also conducted to verify the validity of our proposed baseline.

An Efficient Transformer-Based System for Text-Based Video Segment Retrieval Using FAISS

An efficient system for text-based video segment retrieval is presented, leveraging transformer- based embeddings and the FAISS library for similarity search. The sys- tem enables users to perform real-time, scalable searches over video datasets by converting video segments into combined text and image embeddings. Key components include video segmentation, speech-to-text transcription using Wav2Vec 2.0, frame extraction, embedding generation using Vision Transformers and Sentence Transformers, and efficient similarity search using FAISS. Experimental results demonstrate the system’s applicability in media archives, education, and content discovery, even when applied to a small dataset.

Can artificial intelligence aid the urologists in detecting bladder cancer?

ABSTRACT Introduction: The emergence of artificial intelligence (AI)-based support system endoscopy, including cystoscopy, has shown promising results by training deep learning algorithms with large datasets of images and videos. This AI-aided cystoscopy has the potential to significantly transform the urological practice by assisting the urologists in identifying malignant areas, especially considering the diverse appearance of these lesions. Methods: Four databases, the PubMed, ProQuest, EBSCOHost, and ScienceDirect were searched, along with a manual hand search. Prospective and retrospective studies, experimental studies, cross-sectional studies, and case–control studies assessing the utilization of AI for the detection of bladder cancer through cystoscopy and comparing with the histopathology results as the reference standard were included. The following terms and their variants were used: “artificial intelligence,” “cystoscopy,” and “bladder cancer.” The risk of bias was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. A random effects model was used to calculate the pooled sensitivity and specificity. The Moses–Littenberg model was used to derive the Summary Receiver Operating Characteristics (SROC) curve. Results: Five studies were selected for the analysis. Pooled sensitivity and specificity were 0.953 (95% confidence interval [CI]: 0.908–0.976) and 0.957 (95% CI: 0.923–0.977), respectively. Pooled diagnostic odd ratio was 449.79 (95% CI: 12.42–887.17). SROC curve (area under the curve: 0.988, 95% CI: 0.982–0.994) indicated a strong discriminating power of AI-aided cystoscopy in differentiation normal or benign bladder lesions from the malignant ones. Conclusions: Although the utilization of AI for aiding in the detection of bladder cancer through cystoscopy remains questionable, it has shown encouraging potential for enhancing the detection rates. Future studies should concentrate on identification of the patients groups which could derive maximum benefit from accurate identification of the bladder cancer, such as those with intermediate or high-risk invasive tumors.

Tracking and Behavior Analysis of Group-Housed Pigs Based on a Multi-Object Tracking Approach

Smart farming technologies to track and analyze pig behaviors in natural environments are critical for monitoring the health status and welfare of pigs. This study aimed to develop a robust multi-object tracking (MOT) approach named YOLOv8 + OC-SORT(V8-Sort) for the automatic monitoring of the different behaviors of group-housed pigs. We addressed common challenges such as variable lighting, occlusion, and clustering between pigs, which often lead to significant errors in long-term behavioral monitoring. Our approach offers a reliable solution for real-time behavior tracking, contributing to improved health and welfare management in smart farming systems. First, the YOLOv8 is employed for the real-time detection and behavior classification of pigs under variable light and occlusion scenes. Second, the OC-SORT is utilized to track each pig to reduce the impact of pigs clustering together and occlusion on tracking. And, when a target is lost during tracking, the OC-SORT can recover the lost trajectory and re-track the target. Finally, to implement the automatic long-time monitoring of behaviors for each pig, we created an automatic behavior analysis algorithm that integrates the behavioral information from detection and the tracking results from OC-SORT. On the one-minute video datasets for pig tracking, the proposed MOT method outperforms JDE, Trackformer, and TransTrack, achieving the highest HOTA, MOTA, and IDF1 scores of 82.0%, 96.3%, and 96.8%, respectively. And, it achieved scores of 69.0% for HOTA, 99.7% for MOTA, and 75.1% for IDF1 on sixty-minute video datasets. In terms of pig behavior analysis, the proposed automatic behavior analysis algorithm can record the duration of four types of behaviors for each pig in each pen based on behavior classification and ID information to represent the pigs’ health status and welfare. These results demonstrate that the proposed method exhibits excellent performance in behavior recognition and tracking, providing technical support for prompt anomaly detection and health status monitoring for pig farming managers.

A fully locally selective large kernel network for traffic video detection

Vision Transformer (ViT) are widely used in the field of traffic video inspection because of their wide sensing field and strong global context-capturing capability. However, the ViT model has inadequate local detail feature extraction and high computational complexity. Convoluted neural networks(CNN) are commonly used as a means of video image detection. Although it is more flexible and less computationally complex in the extraction of local details, the fixed-size kernel limits it. It is difficult to effectively utilize the full local context information, resulting in the difficulty of effectively extracting the tiny targets with uneven scale distribution in the region. It is prone to leakage and misdetection of small targets. In view of this, this study proposes a new fully localized selective large kernel network (FL-SLKNet). The network effectively captures specific targets and background information through a fully localized feature extraction method, thus making small targets in the region more distinctive features. On this basis, the Adaptive Expansion Residual Module (AERM) is proposed, which solves the problem of image resolution degradation due to sensory field expansion by combining point-by-point convolution and expansion convolution and utilizing residual connections to supplement the lost information. In order to effectively identify tiny targets in the local area, this study proposes the Multi-Scale Frequency Domain Encoder (MSFDEncoder); this encoder utilizes a high-frequency branch to capture local details and a low-frequency branch to focus on the global structure, effectively capturing granular features of targets at various scales. Experimental results on the VisDrone2019-DET and BDD-100K datasets indicate that FL-SLKNet outperforms other advanced models in traffic video detection, with an increase of 3.3% and 3.4% in mAP0.5 compared to the RT-DETR model. Meanwhile, the detection performance of FL-SLKNet is excellent compared with the YOLO series on the SZ Actual Traffic Video dataset. In addition, in the rainy and hazy simulation scenarios, FL-SLKNet’s mAP0.5 improves by 5.5% and 2.6% compared to the RT-DETR model, and the extreme weather robustness performance is better.

Review on synergizing the Metaverse and AI-driven synthetic data: enhancing virtual realms and activity recognition in computer vision

The Metaverse’s emergence is redefining digital interaction, enabling seamless engagement in immersive virtual realms. This trend’s integration with AI and virtual reality (VR) is gaining momentum, albeit with challenges in acquiring extensive human action datasets. Real-world activities involve complex intricate behaviors, making accurate capture and annotation difficult. VR compounds this difficulty by requiring meticulous simulation of natural movements and interactions. As the Metaverse bridges the physical and digital realms, the demand for diverse human action data escalates, requiring innovative solutions to enrich AI and VR capabilities. This need is underscored by state-of-the-art models that excel but are hampered by limited real-world data. The overshadowing of synthetic data benefits further complicates the issue. This paper systematically examines both real-world and synthetic datasets for activity detection and recognition in computer vision. Introducing Metaverse-enabled advancements, we unveil SynDa’s novel streamlined pipeline using photorealistic rendering and AI pose estimation. By fusing real-life video datasets, large-scale synthetic datasets are generated to augment training and mitigate real data scarcity and costs. Our preliminary experiments reveal promising results in terms of mean average precision (mAP), where combining real data and synthetic video data generated using this pipeline to train models presents an improvement in mAP (32.35%), compared to the mAP of the same model when trained on real data (29.95%). This demonstrates the transformative synergy between Metaverse and AI-driven synthetic data augmentation.

HRTracker: Multi-Object Tracking in Satellite Video Enhanced by High-Resolution Feature Fusion and an Adaptive Data Association

Multi-object tracking in satellite videos (SV-MOT) is an important task with many applications, such as traffic monitoring and disaster response. However, the widely studied multi-object tracking (MOT) approaches for general images can rarely be directly introduced into remote sensing scenarios. The main reasons for this can be attributed to the following: (1) the existing MOT approaches would cause a significant rate of missed detection of the small targets in satellite videos; (2) it is difficult for the general MOT approaches to generate complete trajectories in complex satellite scenarios. To address these problems, a novel SV-MOT approach enhanced by high-resolution feature fusion and a two-step association method is proposed. In the high-resolution detection network, a high-resolution feature fusion module is designed to assist detection by maintaining small object features in forward propagation. By utilizing features of different resolutions, the performance of the detection of small targets in satellite videos is improved. Through high-quality detection and the use of an adaptive Kalman filter, the densely packed weak objects can be effectively tracked by associating almost every detection box instead of only the high-score ones. The comprehensive experimental results using the representative satellite video datasets (VISO) demonstrate that the proposed HRTracker with the state-of-the-art (SOTA) methods can achieve competitive performance in terms of the tracking accuracy and the frequency of ID conversion, obtaining a tracking accuracy score of 74.6% and an ID F1 score of 78.9%.

ASF-Net: Robust video deraining via temporal alignment and online adaptive learning

In recent times, learning-based methods for video deraining have demonstrated commendable results. However, there are two critical challenges that these methods are yet to address: exploiting temporal correlations among adjacent frames and ensuring adaptability to unknown real-world scenarios. To overcome these challenges, we explore video deraining from a paradigm design perspective to learning strategy construction. Specifically, we propose a new computational paradigm, Alignment-Shift-Fusion Network (ASF-Net), which incorporates a temporal shift module. This module is novel to this field and provides deeper exploration of temporal information by facilitating the exchange of channel-level information within the feature space. To fully discharge the model’s characterization capability, we further construct a LArge-scale RAiny video dataset (LARA) which also supports the development of this community. On the basis of the newly-constructed dataset, we explore the parameters learning process by developing an innovative re-degraded learning strategy. This strategy bridges the gap between synthetic and real-world scenes, resulting in stronger scene adaptability. Our approach exhibits superior performance in three benchmarks and compelling visual quality in real-world scenarios, underscoring its efficacy. The code is available at https://github.com/vis-opt-group/ASF-Net.

How attentive is a student in class? A concentration evaluation method based on head Euler angle

How attentive is a student in class? It is an interesting and challenging question in the field of education research. In this paper, a concentration evaluation method based on head Euler angle is proposed. First, a fine large-small-interval video sampling strategy is advocated and concepts of dynamic threshold and action amplitude coefficient are defined, which are fused together to get the head up rate. Second, head front-facing rate is computed through analyzing the yaw angle, which is obtained from a proposed spatial physical model of camera and student seat position. Third, for any given time period, class concentration of a student is derived by fusing and normalizing the head up rate and head front-facing rate based on Euclidean-distance. Finally, accuracy experiments are conducted, where the proposed method achieves an average accuracy of 89.8% compared with the average score from the questionnaire given by 22 reviewers. Also, robust experiments on several head pose estimation models are facilitated, which achieves an average accuracy of 98.43% on recorded video dataset, indicates insensitivity to the head pose estimation model used for data collection and verifies that the proposed method is independent on any specific head pose estimation model. The experimental results show that the proposed method is accurate, effective and robust for the concentration evaluation in class.

Rapeseed Flower Counting Method Based on GhP2-YOLO and StrongSORT Algorithm.

Accurately quantifying flora and their respective anatomical structures within natural ecosystems is paramount for both botanical breeders and agricultural cultivators. For breeders, precise plant enumeration during the flowering phase is instrumental in discriminating genotypes exhibiting heightened flowering frequencies, while for growers, such data inform potential crop rotation strategies. Moreover, the quantification of specific plant components, such as flowers, can offer prognostic insights into the potential yield variances among different genotypes, thereby facilitating informed decisions pertaining to production levels. The overarching aim of the present investigation is to explore the capabilities of a neural network termed GhP2-YOLO, predicated on advanced deep learning techniques and multi-target tracking algorithms, specifically tailored for the enumeration of rapeseed flower buds and blossoms from recorded video frames. Building upon the foundation of the renowned object detection model YOLO v8, this network integrates a specialized P2 detection head and the Ghost module to augment the model's capacity for detecting diminutive targets with lower resolutions. This modification not only renders the model more adept at target identification but also renders it more lightweight and less computationally intensive. The optimal iteration of GhP2-YOLOm demonstrated exceptional accuracy in quantifying rapeseed flower samples, showcasing an impressive mean average precision at 50% intersection over union metric surpassing 95%. Leveraging the virtues of StrongSORT, the subsequent tracking of rapeseed flower buds and blossom patterns within the video dataset was adeptly realized. By selecting 20 video segments for comparative analysis between manual and automated counts of rapeseed flowers, buds, and the overall target count, a robust correlation was evidenced, with R-squared coefficients measuring 0.9719, 0.986, and 0.9753, respectively. Conclusively, a user-friendly "Rapeseed flower detection" system was developed utilizing a GUI and PyQt5 interface, facilitating the visualization of rapeseed flowers and buds. This system holds promising utility in field surveillance apparatus, enabling agriculturalists to monitor the developmental progress of rapeseed flowers in real time. This innovative study introduces automated tracking and tallying methodologies within video footage, positioning deep convolutional neural networks and multi-target tracking protocols as invaluable assets in the realms of botanical research and agricultural administration.

Review Paper on Deep RL for Video Summarization

In recent years, the explosive growth of video content has heightened the need for efficient summarization techniques to distill lengthy videos into concise, informative summaries. Traditional approaches to video summarization often rely on heuristic-based methods or supervised learning techniques, which can be limited by their reliance on predefined features or extensive labeled datasets. To address these limitations, this paper explores the application of Deep Reinforcement Learning (DRL) for video summarization. DRL offers a dynamic framework where an agent learns to optimize summarization strategies through interaction with the video content, enabling adaptive and context-aware summarization. We propose a novel DRL-based framework that combines convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to extract and represent temporal features from video frames. Actor- critic architecture is utilized, where the actor generates candidate summaries and the critic evaluates their quality based on a reward function designed to balance in formativeness and brevity. We introduce a new reward function that incorporates both content relevance and diversity to encourage the generation of summaries that effectively capture key moments and maintain narrative coherence. Experimental results on benchmark video datasets demonstrate that our DRL-based approach significantly outperforms traditional methods in terms of both summary quality and user satisfaction. The proposed method not only achieves state-of-the-art performance but also offers greater flexibility and adaptability to diverse video content. This work highlights the potential of DRL in advancing video summarization and opens avenues for future research in optimizing video content extraction and representation

A joint time and spatial attention-based transformer approach for recognizing the behaviors of wild giant pandas

Wild giant pandas, an endangered species exclusive to China, are a focus of conservation efforts. The behavior of giant pandas reflects their health conditions and activity capabilities, which play an important role in formulating and implementing conservation measures. Researching and developing efficient behavior recognition methods based on deep learning can significantly advance the study of wild giant panda behavior. This study introduces, for the first time, a transformer-based behavior recognition method termed PandaFormer, which employs time-spatial attention to analyze behavioral temporal patterns and estimate activity spaces. The method integrates advanced techniques such as cross-fusion recurrent time encoding and transformer modules, which handle both temporal dynamics and spatial relationships within panda behavior videos. First, we design cross-fusion recurrent time encoding to represent the occurrence time of behaviors effectively. By leveraging the multimodal processing capability of the transformer, we input time and video tokens into the transformer module to explore the relation between behavior and occurrence time. Second, we introduce relative temporal weights between video frames to enable the model to learn sequential relationships. Finally, considering the fixed position of the camera during recording, we propose a spatial attention mechanism based on the estimation of the panda's activity area. To validate the effectiveness of the model, a video dataset of wild giant pandas, encompassing five typical behaviors, was constructed. The proposed method is evaluated on this video-level annotated dataset. It achieves a Top-1 accuracy of 92.25 % and a mean class precision of 91.19 %, surpassing state-of-the-art behavior recognition algorithms by a large margin. Furthermore, the ablation experiments validate the effectiveness of the proposed temporal and spatial attention mechanisms. In conclusion, the proposed method offers an effective way of studying panda behavior and holds potential for application to other wildlife species.

Just Dance: detection of human body reenactment fake videos

In the last few years, research on the detection of AI-generated videos has focused exclusively on detecting facial manipulations known as deepfakes. Much less attention has been paid to the detection of artificial non-facial fake videos. In this paper, we address a new forensic task, namely, the detection of fake videos of human body reenactment. To this purpose, we consider videos generated by the “Everybody Dance Now” framework. To accomplish our task, we have constructed and released a novel dataset of fake videos of this kind, referred to as FakeDance dataset. Additionally, we propose two forgery detectors to study the detectability of FakeDance kind of videos. The first one exploits spatial–temporal clues of a given video by means of hand-crafted descriptors, whereas the second detector is an end-to-end detector based on Convolutional Neural Networks (CNNs) trained on purpose. Both detectors have their peculiarities and strengths, working well in different operative scenarios. We believe that our proposed dataset together with the two detectors will contribute to the research on the detection of non-facial fake videos generated by means of AI.

Scene adaptation in adverse conditions: a multi-sensor fusion framework for roadside traffic perception

Robust roadside traffic perception requires integrating the strengths of multi-source sensors under various adverse conditions, which is challenging but indispensable for formulating effective traffic management strategies. One limitation of existing radar-camera perception systems is that they focus on integrating multi-source information without directly considering scene information, leading to difficulties in achieving scene adaptive fusion. How to establish the connection between scene information and multi-source information is the key challenge to solving this problem. In this article, we propose a Scene adaptive Sensor Fusion (SSF) framework that characterizes scene information and integrates it into radar-camera fusion schemes, aiming to achieve high-quality roadside traffic perception. Specifically, we introduce a multi-source object association method that accurately associates multi-source sensor information on the roadside. We then utilize coding techniques to characterize the scene information, including visibility characterization regarding lighting and weather conditions, and road characterization regarding sensor viewpoint. By incorporating sensor and scene information into the fusion model, the SSF framework effectively establishes the connection between them. We evaluate the SSF framework on the Roadside Radar and Video Dataset (RRVD) and the Traffic flow Parameter Estimation Dataset (TPED), both collected from real-world traffic scenarios. Experiments demonstrate that SSF significantly improves vehicle detection accuracy under various adverse conditions compared to traditional single-source sensing methods and other state-of-the-art fusion techniques. Furthermore, vehicle trajectories based on SSF detection results enable accurate traffic parameter estimation, such as volume, speed, and density, in complex and dynamic environments.

Noise robustness evaluation of image processing algorithms for eye blinking detection

Robust algorithms for eye blinking detection are required due to the effects of noisy environments and varying light conditions on image-based detection methods. This paper compares five non-supervised image-based algorithms for eye blinking detection, evaluating their robustness to additive Gaussian noise. The algorithms were tested on a video dataset acquired using a smartphone and an ophthalmology chin rest. Through Monte Carlo simulation that introduces Gaussian noise at different intensities, we evaluate the algorithms’ precision, sensitivity, and F1-scores for frame classification, True Positive Rate (TPR), False Discovery Rate (FDR) and F1-score for the event detector. The results of experimental tests reveal significant variations in algorithms’ performance with increasing noise levels. Notably, the Image Correlation (IC) algorithm demonstrates superior eye blinking detection capabilities under various noise conditions, emerging as the most robust algorithm among those tested. This distinction highlights the potential of IC for reliable blink detection in noisy environments.

Human pose estimation in complex background videos via Transformer-based multi-scale feature integration

Human posture estimation is still a hot research topic. Previous algorithms based on traditional machine learning have difficulties in feature extraction and low fusion efficiency. To address these problems, we proposed a Transformer-based method. We combined three techniques, namely the Transformer-based feature extraction module, the multi-scale feature fusion module, and the occlusion processing mechanism, to capture the human pose. The Transformer-based feature extraction module uses the self-attention mechanism to extract key features from the input sequence, the multi-scale feature fusion module fuses feature information of different scales to enhance the perception ability of the model, and the occlusion processing mechanism can effectively handle occlusion in the data and effectively remove background interference. Our method has shown excellent performance through verification on the standard dataset Human3.6M and the wild video dataset, achieving accurate pose prediction in both complex actions and challenging samples.

Interobserver agreement of an ED PoCUS video training dataset of normal appendix and appendicitis in children

BackgroundEducational video datasets can be an effective method for training in emergency department (ED) point-of-care ultrasound (PoCUS). A video dataset for normal appendix and appendicitis in children using ED PoCUS images was developed to assess interobserver agreement, as measured by Cohen’s Kappa on key sonographic findings.MethodsThree sets of 25 ED PoCUS videos were selected and curated from pediatric patients with normal appendix and acute appendicitis. Four participant ED sonologist-physicians were trained on the first set of 25 videos showing normal appendix or normal bowel in patients without appendicitis to note if normal appendix was seen in any part or in it’s entirety from tip-to-cecum. They were then tested on the second set of similar videos. A third set of 25 videos from patients who had appendicitis where participant sonologists were asked to note if appendicitis was present or absent, with and without appendicolith or perforation. Cohen’s Kappa was calculated in aggregate and stratified by experience vs. novice against a senior sonologist-physician aware of all patient outcomes for visualization of: 1. any part of normal appendix, 2. normal appendix visualized from tip to cecum 3. any part of appendicitis, 4. appendicolith, 5. appendiceal perforation.ResultsCohen’s Kappa for any part of normal appendix, 0.71, 95% CI (0.58–0.85); normal appendix tip-to-cecum, 0.43, 95% CI (0.19–0.67), appendicitis, 0.53, 95%CI (0.34–0.70), appendicolith, 0.63, 95%CI (0.43–0.84), perforated appendicitis, 0.46, 95%CI (0.22–0.70). Stratified by experienced vs. novice: any part of normal appendix, 0.75 vs. 0.68; normal appendix tip-to-cecum, 0.50 vs. 0.36; appendicitis, 0.78 vs. 0.31; appendicolith, 0.75 vs. 0.5; perforated appendicitis, 0.5 vs 0.42.ConclusionsThis educational video dataset may be used to train sonologist-physicians in ED PoCUS scanning for normal appendix and appendicitis in children. Sonologist experience affected interobserver agreement with respect to visualization of entire normal appendix and appendicitis.

Fitcam: detecting and counting repetitive exercises with deep learning

Physical fitness is one of the most important traits a person could have for health longevity. Conducting regular exercise is fundamental to maintaining physical fitness, but with the caveat of occurring injury if not done properly. Several algorithms exists to automatically monitor and evaluate exercise using the user’s pose. However, it is not an easy task to accurately monitor and evaluate exercise poses automatically. Moreover, there are limited number of datasets exists in this area. In our work, we attempt to construct a neural network model that could be used to evaluate exercise poses based on key points extracted from exercise video frames. First, we collected several images consists of different exercise poses. We utilize the the OpenPose library to extract key points from exercise video datasets and LSTM neural network to learn exercise patterns. The result of our experiment has shown that the methods used are quite effective for exercise types of push-up, sit-up, squat, and plank. The neural-network model achieved more than 90% accuracy for the four exercise types.

Tiny Object Tracking: A Large-Scale Dataset and a Baseline.

Tiny objects, frequently appearing in practical applications, have weak appearance and features, and receive increasing interests in many vision tasks, such as object detection and segmentation. To promote the research and development of tiny object tracking, we create a large-scale video dataset, which contains 434 sequences with a total of more than 217K frames. Each frame is carefully annotated with a high-quality bounding box. In data creation, we take 12 challenge attributes into account to cover a broad range of viewpoints and scene complexities, and annotate these attributes for facilitating the attribute-based performance analysis. To provide a strong baseline in tiny object tracking, we propose a novel multilevel knowledge distillation network (MKDNet), which pursues three-level knowledge distillations in a unified framework to effectively enhance the feature representation, discrimination, and localization abilities in tracking tiny objects. Extensive experiments are performed on the proposed dataset, and the results prove the superiority and effectiveness of MKDNet compared with state-of-the-art methods. The dataset, the algorithm code, and the evaluation code are available at https://github.com/mmic-lcl/Datasets-and-benchmark-code.