Video Frames Research Articles

Snapshot computed tomographic microscopic imaging spectrometer and its video-level tracking of poisonous Microcystis aeruginosa cells in mixed algae

Conventional microscopic spectral imaging suffers from extended scanning times across wavelength or spatial dimension. To improve capabilities of dynamic microscopic spectral imaging, we developed a snapshot computed tomographic microscopic imaging spectrometer (CTMIS) based on the zeroth and first orders dispersive diffraction of a two-dimensional grating. Utilizing the CTMIS-UNET reconstruction algorithm, we can reconstruct a spectral cube (541x541x26) for each frame of micro spectral imaging video. Experimental results demonstrate a sub-4 μm spatial resolution achievable through a 20x objective lens and a spectral resolution better than 10 nm among 450–700 nm, while maintaining spectral cosine similarities exceeding 0.9989 when comparing reconstructed spectra with ground truth data. Spectral imaging videos of four species of algae and mixed algae were captured under 10 ms exposure time using the CTMIS system. Leveraging the self-developed UNET-SI26 algae recognition network, precise identification and tracking of four types of algae and poisonous microcysts aeruginosa in mixed algae were conducted. The pixel-level recognition accuracy exceeds 95 %, while the accuracy for counting the numbers of different types of cells surpasses 85 %, offering an efficient and accurate spectral imaging method for real-time monitoring and early warning of harmful algae at the cellular level.

Automatic Identification of Game Stuttering via Gameplay Videos Analysis

Modern video games are extremely complex software systems and, as such, they might suffer from several types of post-release issues. A particularly insidious issue is constituted by drops in the frame rate ( i.e. , stuttering events), which might have a negative impact on the user experience. Stuttering events are frequently documented in the million of hours of gameplay videos shared by players on platforms such as Twitch or YouTube. From the developers’ perspective, these videos represent a free source of documented “testing activities”. However, especially for popular games, the quantity and length of these videos make impractical their manual inspection. We introduce HASTE, an approach for the automatic detection of stuttering events in gameplay videos that can be exploited to generate candidate bug reports. HASTE firstly splits a given video into visually coherent slices, with the goal of filtering-out those that not representing actual gameplay ( e.g ., navigating the game settings). Then, it identifies the subset of pixels in the video frames which actually show the game in action excluding additional elements on screen such as the logo of the YouTube channel, on-screen chats etc. In this way, HASTE can exploit state-of-the-art image similarity metrics to identify candidate stuttering events, namely subsequent frames being almost identical in the pixels depicting the game. We evaluate the different steps behind HASTE on a total of 105 videos showing that it can correctly extract video slices with a 76% precision, and can correctly identify the slices related to gameplay with a recall and precision higher than 77%. Overall, HASTE achieves 71% recall and 89% precision for the identification of stuttering events in gameplay videos.

SimpleTrackV2: Rethinking the Timing Characteristics for Multi-Object Tracking.

Multi-object tracking tasks aim to assign unique trajectory codes to targets in video frames. Most detection-based tracking methods use Kalman filtering algorithms for trajectory prediction, directly utilizing associated target features for trajectory updates. However, this approach often fails, with camera jitter and transient target loss in real-world scenarios. This paper rethinks state prediction and fusion based on target temporal features to address these issues and proposes the SimpleTrackV2 algorithm, building on the previously designed SimpleTrack. Firstly, to address the poor prediction performance of linear motion models in complex scenes, we designed a target state prediction algorithm called LSTM-MP, based on long short-term memory (LSTM). This algorithm encodes the target's historical motion information using LSTM and decodes it with a multilayer perceptron (MLP) to achieve target state prediction. Secondly, to mitigate the effect of occlusion on target state saliency, we designed a spatiotemporal attention-based target appearance feature fusion (TSA-FF) target state fusion algorithm based on the attention mechanism. TSA-FF calculates adaptive fusion coefficients to enhance target state fusion, thereby improving the accuracy of subsequent data association. To demonstrate the effectiveness of the proposed method, we compared SimpleTrackV2 with the baseline model SimpleTrack on the MOT17 dataset. We also conducted ablation experiments on TSA-FF and LSTM-MP for SimpleTrackV2, exploring the optimal number of fusion frames and the impact of different loss functions on model performance. The experimental results show that SimpleTrackV2 handles camera jitter and target occlusion better, achieving improvements of 1.6%, 3.2%, and 6.1% in MOTA, IDF1, and HOTA, respectively, compared to the SimpleTrack algorithm.

Mask R-CNN and Centroid Tracking Algorithm to Process UAV Based Thermal-RGB Video for Drylot Cattle Heat Stress Monitoring

This study developed and evaluated an algorithm for processing thermal-RGB video feeds captured by an unmanned aerial vehicle (UAV) to automate heat stress monitoring in cattle housed in the drylots. The body surface temperature (BST) of individual cows was used as an indicator of heat stress. UAV data were collected using RGB and thermal infrared imagers, respectively, at 2 and 6.67 cm per pixel spatial resolution in Spring 2023 (dataset-1) and Summer 2024 (dataset-2). Study sites were two commercial drylots in Washington State. The custom algorithms were developed to: (1) detect and localize individual cows using a Mask R-CNN-based instance segmentation model combined with centroid tracking; and (2) extract BST by averaging the thermal-imagery pixels for each of the segmented cows. The algorithm showed higher detection accuracy with RGB images as input (F1 score: 0.89) compared to thermal (F1 score: 0.64). BST extraction with combined RGB and thermal imaging approach required corrections for alignment problems associated with differences in optics, imaging field of view, resolution, and lens properties. Consequently, thermal imaging-only approach was adopted for assessing real-time cow localization and BST estimation. Operating at one frame per second, algorithm successfully detected 72.4% and 81.65% of total cows in video frames from dataset-1 (38 s) and -2 (48 s), respectively. The mean absolute difference between algorithm output and ground truth (BSTGT) was 2.1 °C (dataset-1) and 3.3 °C (dataset-2), demonstrating satisfactory performance. With further refinements, this approach could be a viable tool for real-time heat stress monitoring in large-scale drylot production systems.

What physical examinations are observed during an in-person GP consultation? Automatic extraction using a text-based approach

ObjectivesTeleconsultation is anticipated to have a long-term role in primary care. However, conducting virtual physical examinations is a well-known limitation. To anticipate unmet needs general practitioners (GPs) and patients may experience during teleconsultation, this study aims to automatically identify physical examinations typically conducted during in-person GP consultation. Material and MethodsThis study utilizes 281 GP in-person consultations (de-identified transcripts & video recordings) within UK general practices, where 169 eligible ones were included in this study. We propose an automated text-based approach using regular expressions on keywords in GP-patient consultation dialogue (e.g., “roll up your sleeves”) to identify physical examinations (e.g. blood pressure measurement). This approach involves the construction of conceptual diagrams to visually inspect the relationship between keywords and physical examinations, syntax analysis to identify patterns between keywords and generate regular expressions, and the use of these regular expressions in consultation transcripts to detect potential instances of physical examinations, where matching video frames were subsequently retrieved. The performance of our automated text-based approach is compared to manual classification by 2 independent researchers using 5-fold cross-validation (precision, recall, and F1-score). ResultsAmong the 169 eligible GP in-person consultations, 133 (79%) required a physical examination, while the other 33 visits were for psychological reasons. Out of these 133 consultations, a total of 283 physical examinations were observed, with 21 instances conducted behind a curtain. We identified 42 distinct types of physical examinations from these 283 instances, grouped into 10 physical examination categories based on body areas and physical artefacts. The most frequent category of physical examinations is Vital Signs 26.80% (76/283). Overall, blood pressure measurement (also belonging to the Vital Signs category) is the most frequent physical examination at 59.2% (45/76). The comparison between manual classification and the regular expression model demonstrates an average precision of 88.3%, recall of 78.9%, and an F1-score of 83.3% from 5-fold cross-validation, providing significant insights into the frequency and types of physical examinations conducted during in-person GP consultations. ConclusionBy using regular expressions in consultation dialogues between GPs and patients, we can automatically identify physical examinations in GP consultations with a precision of 88.3%. Findings from this study, i.e. physical examinations during in-person GP consultations, provide insights into areas where GPs and patients may need support during teleconsultation.

Video Anomaly Detection via self-supervised and spatio-temporal proxy tasks learning

Video Anomaly Detection (VAD) aims to identify events in videos that deviate from typical patterns. Given the scarcity of anomalous samples, previous research has primarily focused on learning regular patterns from datasets exclusively containing normal behaviors, and treating deviations from these patterns as anomalies. However, most of these methods are constrained by coarse-grained modeling approaches that renders them incapable of learning highly-discriminative features, which are necessary to effectively distinguish between the subtle differences between normal and abnormal behaviors. To better capture these features, we propose an innovative method. Initially, pseudo-anomalous samples for appearance and motion are generated through geometric transformations (2D rotations) and the scrambling of video sequences. Subsequently, a dual-branch network featuring spatio-temporal decoupling is proposed, in which the spatial and temporal branches each handle a specific proxy task. These tasks are designed to distinguish between normal and pseudo-anomalous samples, involving operations such as predicting patch-based 2D rotation angles and classifying video frame triplets as total-anomaly, left-anomaly, right-anomaly, and non-anomaly. Our approach employs an end-to-end training methodology, without relying on pre-trained models (except for the object detector). Evaluations on the UCSD Ped2, Avenue, and ShanghaiTech datasets show that our method achieved AUC scores of 99.1%, 91.9%, and 81.1%, respectively, demonstrating its effectiveness. The code is publicly accessible at the following link: https://spatio-temporal-tasks.

Behavior recognition of cage-free multi-broilers based on spatiotemporal feature learning

Poultry behavior indicates their health, welfare, and production performance. Timely access to broilers' behavioral information can improve their welfare and reduce disease spread. Most behaviors require a period of observation before they can be accurately judged. However, the existing approaches for multi-object behavior recognition were mostly developed based on a single-frame image and ignored the temporal features in videos, which led to misrecognition. This study proposed an end-to-end method for recognizing multiple simultaneous behavioral events of cage-free broilers in videos by Broiler Behavior Recognition System (BBRS) based on spatiotemporal feature learning. The BBRS consisted of 3 main components: the improved YOLOv8s detector, the Bytetrack tracker, and the 3D-ResNet50-TSAM model. The basic network YOLOv8s was improved with MPDIoU to identify multiple broilers in the same frame of videos. The Bytetrack tracker was used to track each identified broiler and acquire its image sequence of 32 continuous frames as input for the 3D-ResNet50-TSAM model. To accurately recognize behavior of each tracked broiler, the 3D-ResNet50-TSAM model integrated a temporal-spatial attention module for learning the spatiotemporal features from its image sequence and enhancing inference ability in the case of its image sequence less than 32 continuous frames due to its tracker ID switching. Each component of BBRS was trained and tested with the rearing density of 7 to 8 birds/m2. The results demonstrated that the mAP@0.5 of the improved YOLOv8s detector was 99.50%. The Bytetrack tracker achieved a mean MOTA of 93.89% at different levels of occlusion. The Accuracy, Precision, Recall, and F1score of the 3D-ResNet50-TSAM model were 97.84, 97.72, 97.65, and 97.68%, respectively. The BBRS showed satisfactory inference ability with an Accuracy of 93.98% when 26 continuous frames of the tracked broiler were received by the 3D-ResNet50-TSAM model. This study provides an efficient tool for automatically and accurately recognizing behaviors of cage-free multi-broilers in videos. The code will be released on GitHub (https://github.com/CoderYLH/BBRS) as soon as the study is published.

Video frame interpolation neural network for 3D tomography across different length scales

Three-dimensional (3D) tomography is a powerful investigative tool for many scientific domains, going from materials science, to engineering, to medicine. Many factors may limit the 3D resolution, often spatially anisotropic, compromising the precision of the information retrievable. A neural network, designed for video-frame interpolation, is employed to enhance tomographic images, achieving cubic-voxel resolution. The method is applied to distinct domains: the investigation of the morphology of printed graphene nanosheets networks, obtained via focused ion beam-scanning electron microscope (FIB-SEM), magnetic resonance imaging of the human brain, and X-ray computed tomography scans of the abdomen. The accuracy of the 3D tomographic maps can be quantified through computer-vision metrics, but most importantly with the precision on the physical quantities retrievable from the reconstructions, in the case of FIB-SEM the porosity, tortuosity, and effective diffusivity. This work showcases a versatile image-augmentation strategy for optimizing 3D tomography acquisition conditions, while preserving the information content.

A new method of intraoperative assessment of the dynamics of cortical blood flow using imaging photoplethysmography

Background. Intraoperative assessment of changes in cerebral blood flow is an important component of objective quality control of surgical treatment of cerebral artery aneurysms. Various techniques have been tried to solve this task, but they all have their drawbacks, which forces us to look for new ways of blood flow monitoring. We propose to use the technology of imaging photoplethysmography (IPPG) – a technically simple, contactless, safe and cheap optical method for assessing the perfusion of biological tissues.Aim. To demonstrate the possibility of using IPPG to assess the dynamics of cerebral blood flow parameters during aneurysm clipping surgery, as well as to identify early changes in blood supply to the cerebral cortex.Materials and methods. The study was carried out during six surgeries of clipping aneurysms of the anterior part of the Willis’s circle, both in the acute stage of rupture (n = 1) and in a planned manner (n = 5). The IPPG system, which is an LED illuminator in a single unit with a digital video camera, was located on a tripod 25 cm from the intervention zone. During each operation, two one-minute recordings of the illuminated surface of the cerebral cortex were performed: after dissection of the dura mater and before its suturing at the end of the main stage of the intervention. To improve the measurement accuracy, video frames of the studied area were recorded synchronously with the registration of an electrocardiogram. After recording, two IPPG parameters were calculated and compared: the amplitude of the pulsatile component and the pulse wave transit time. Thereafter, the obtained data were compared with the results of computed tomography. Statistical analysis was performed using pairwise comparison tests in the GraphPad Prism software package.Results. Clipping of cerebral vessel aneurysms are accompanied by significant changes in the parameters of cerebral blood flow. Analysis of the data for all patients revealed significant differences in IPPG parameters before and after surgery, namely, statistically significant increase in amplitude of the pulsatile component (n = 3) and decrease in pulse wave transit time (n = 5). The absence of significant changes in both parameters was found only in one patient who had mechanical damage in the cortex in the region of video recording.Conclusion. The IPPG system is capable to quantify changes in blood supply to the cortex during surgical treatment of cerebral artery aneurysms and to identify areas with either increased or decreased blood supply. In-depth studies are required to obtain additional markers of the postoperative state of cerebral blood flow.

Fishing event detection and species classification using computer vision and artificial intelligence for electronic monitoring

Fisheries regulations require detailed catch reporting on commercial fishing vessels. Vital components for the sustainable management of fish stocks include a robust estimate of the number of fish caught and the species composition. Catch recording is often done manually by human observers on fishing vessels. Human observers are costly, and consistent data streams can be subject to observer availability and the weather. On-vessel cameras (electronic monitoring, EM) are a growing alternative to human observers. However, on-land human auditors are required to review hundreds of hours of videos recorded during fishing trips that can last for weeks. In this paper, a framework is presented to automatically detect fish in EM videos, count the total fishing events, and classify the fish species. For this purpose, a deep learning and computer vision-based model is developed to efficiently detect fish and fishers onboard a vessel. Secondly, a vision-based tracking pipeline tracks the detected fish and counts the total fishing events in the videos. Thirdly, the extracted fishing events are classified through a deep learning-based fish species classifier, to provide the distribution of different fish species caught for a fishing trip. For our experiments, the datasets were prepared using the electronic monitoring data of multiple fishing trips of a fishing vessel. The videos were recorded on Australian longline vessels targeting tunas and billfish. For the fish detection task, video frames were extracted and labelled manually to provide a digital ground-truth. For the fish species classification task, hundreds of fish images of multiple species were cropped to provide a training dataset for the fish classifier. For the fish counting task, manual counts for the fishing events of individual fish species were generated for the test fishing trips. The developed fish and fisher detector achieves a mean Average Precision of 87.0 % for fish and 94.0 % for fishers on test video frames. The fishing event detection pipeline achieves an Average Precision of 81.0 % and an Average Recall of 74.5 % on test videos. The fish species classifier achieves an Accuracy (Top-1) of 91.11 % for the classification of cropped fish images and 89.05 % for the classification of extracted fishing events from the videos. Experimental results show that our proposed computer vision and artificial intelligence-based solution for video analysis has great potential to automate the auditing process from electronic monitoring footage and contribute to the sustainable management of fish stocks.

Automated Analysis of Stereotypical Movements in Videos of Children With Autism Spectrum Disorder

Stereotypical motor movements (SMMs) are a form of restricted and repetitive behavior, which is a core symptom of autism spectrum disorder (ASD). Current quantification of SMM severity is extremely limited, with studies relying on coarse and subjective caregiver reports or laborious manual annotation of short video recordings. To assess the utility of a new open-source AI algorithm that can analyze extensive video recordings of children and automatically identify segments with heterogeneous SMMs, thereby enabling their direct and objective quantification. This retrospective cohort study included 241 children (aged 1.4 to 8.0 years) with ASD. Video recordings of 319 behavioral assessments carried out at the Azrieli National Centre for Autism and Neurodevelopment Research in Israel between 2017 and 2021 were extracted. Behavioral assessments included cognitive, language, and autism diagnostic observation schedule, 2nd edition (ADOS-2) assessments. Data were analyzed from October 2020 to May 2024. Each assessment was recorded with 2 to 4 cameras, yielding 580 hours of video footage. Within these extensive video recordings, manual annotators identified 7352 video segments containing heterogeneous SMMs performed by different children (21.14 hours of video). A pose estimation algorithm was used to extract skeletal representations of all individuals in each video frame and was trained an object detection algorithm to identify the child in each video. The skeletal representation of the child was then used to train an SMM recognition algorithm using a 3 dimensional convolutional neural network. Data from 220 children were used for training and data from the remaining 21 children were used for testing. Among 319 behavioral assessment recordings from 241 children (172 [78%] male; mean [SD] age, 3.97 [1.30] years), the algorithm accurately detected 92.53% (95% CI, 81.09%-95.10%) of manually annotated SMMs in our test data with 66.82% (95% CI, 55.28%-72.05%) precision. Overall number and duration of algorithm-identified SMMs per child were highly correlated with manually annotated number and duration of SMMs (r = 0.8; 95% CI, 0.67-0.93; P < .001; and r = 0.88; 95% CI, 0.74-0.96; P < .001, respectively). This study suggests the ability of an algorithm to identify a highly diverse range of SMMs and quantify them with high accuracy, enabling objective and direct estimation of SMM severity in individual children with ASD.

A Mobile Surveillance System with Face Detection and Recognition Abilities

This paper proposes a mobile surveillance system similar to an electronic dog which has the ability to move around a certain location all while detecting and recognizing the individuals and notifying the respective authorities about the detected person. The system uses an Arduino UNO microcontroller along with IR receiver and remote to function as a remote-controlled vehicle while also providing an option for automatic movement of the vehicle. The system also has an integrated camera that constantly feeds live video to a computer system. Haar-cascades take in the frames of videos and detect appearance of human face. Then, pretrained FaceNet Convolutional Neural Network (CNN) compares the detected face to the authorized faces in an attempt to recognize the face. The system also boasts the feature of email notification for which it employs the SMTP protocol to send an email with the detected face to the authorized personnel. The methods and techniques used to implement these features are presented in depth in this paper.

Digital Otoscopy With Computer-Aided Composite Image Generation: Impact on the Correct Diagnosis, Confidence, and Time.

This study investigated the comparative performance of ear, nose, and throat (ENT) physicians in correctly detecting ear abnormalities when reviewing digital otoscopy imaging using 3 different visualization methods, including computer-assisted composite images called "SelectStitch," single video frame "Still" images, and video clips. The study also explored clinicians' diagnostic confidence levels and the time to make a diagnosis. Clinician diagnostic reader study. Online diagnostic survey of ENT physicians. Nine ENT physicians reviewed digital otoscopy examinations from 86 ears with various diagnoses (normal, perforation, retraction, middle ear effusion, tympanosclerosis). Otoscopy examinations used artificial-intelligence (AI)-based computer-aided composite image generation from a video clip (SelectStitch), manually selected best still frame from a video clip (Still), or the entire video clip. Statistical analyses included comparisons of ability to detect correct diagnosis, confidence levels, and diagnosis times. The ENT physicians' ability to detect ear abnormalities (33.2%-68.7%) varied depending on the pathologies. SelectStitch and Still images were not statistically different in detecting abnormalities (P > .50), but both were different from Video (P < .01). However, the performance improvement observed with Videos came at the cost of significantly longer time to determining the diagnosis. The level of confidence in the diagnosis was positively associated with correct diagnoses, but varied by particular pathology. This study explores the potential of computer-assisted techniques like SelectStitch in enhancing otoscopic diagnoses and time-saving, which could benefit telemedicine settings. Comparable performance between computer-generated and manually selected images suggests the potential of AI algorithms for otoscopy applications.

Safeguards-related event detection in surveillance video using semi-supervised learning approach

We develop a deep learning model employing a semi-supervised learning approach, which can detect automatically safeguards-related events in nuclear facility from surveillance video. Our model is designed after a comprehensive analysis of the trends in artificial intelligence-based models to identify abnormal events in video. Our model incorporates a reconstruction module and a prediction module independently. The reconstruction module is trained to generate video frames within a sliding window, while the prediction module is trained to predict future motion feature based on the motion features within the video frames in a sliding window. Each module utilizes an autoencoder with a memory module positioned between an encoder and an decoder of the autoencoder. We evaluate the model's performance using a benchmark dataset and a self-produced dataset obtained from facility related to pyroprocessing. Our model's performanace is comparable to or superior to that of the prevous models from the benchmark dataset analysis, and all the abnormal events can be detected without false positive error from the self-produced dataset analysis.

Motion focus global–local network: Combining attention mechanism with micro action features for cow behavior recognition

The use of machine vision technology for recognizing cow behavior plays a crucial role in daily management, health monitoring, and breeding and reproduction in dairy farming, making it an essential component of modern smart agriculture. This paper presents a novel dual-stream network model, the Motion Focus Global-Local Network (MFGN), for analyzing cow video data. The dual-stream network consists of a global spatiotemporal feature stream and a fine motion feature stream. The global spatiotemporal feature stream extracts key frames to remove redundant information and utilizes a Transformer network for global spatio-temporal feature extraction, reflecting the dynamic changes in cow videos and the temporal relationships between video frames. The fine motion feature stream, based on frame differencing of cow videos, uses focal convolution to capture subtle movements of cows, enhancing the focus on minor behavioral changes. To evaluate the performance of the proposed model, video data samples were collected from eight cows marked on their bodies and heads at an Australian farm site (CSIRO Armidale), including a total of 1715 video sequences across three behavior categories. The model achieved recognition accuracies of 98.1% for drinking, 95.5% for grazing, and 49.3% for other behaviors, with an overall average recognition accuracy of 79.4%, representing a 7.4% improvement over the classic TSN model. Overall, the MFGN network effectively extracts and integrates global spatiotemporal features with fine motion features from cow video data, modeling both the overall sequence characteristics and focusing on local motion details, achieving precise behavioral recognition. This research not only enhances the accuracy of cow behavior recognition but also provides new technological means for precise management in modern smart agriculture, with broad industry application potential to improve farm efficiency, profitability, and disease control and prevention.

RF‐GCN: Residual fused‐graph convolutional network using multimodalities for facial emotion recognition

AbstractBackgroundThe emotional state of individuals is difficult to identify and it is developing now a days because of vast interest in recognition. Many technologies have been developed to identify this emotional expression based on facial expressions, vocal expressions, physiological signals, and body expressions. Among these, facial emotion is very expressive for recognition using multimodalities. Understanding facial emotions has applications in mental well‐being, decision‐making, and even social change, as emotions play a crucial role in our lives. This recognition is complicated by the high dimensionality of data and non‐linear interactions across modalities. Moreover, the way emotion is expressed by people varies and these feature identification remains challenging, where these limitations are overcome by Deep learning models.MethodsThis research work aims at facial emotion recognition through the utilization of a deep learning model, named the proposed Residual Fused‐Graph Convolution Network (RF‐GCN). Here, multimodal data included is video as well as an Electroencephalogram (EEG) signal. Also, the Non‐Local Means (NLM) filter is used for pre‐processing input video frames. Here, the feature selection process is carried out using chi‐square, after feature extraction, which is done in both pre‐processed video frames and input EEG signals. Finally, facial emotion recognition and its types are determined by RF‐GCN, which is a combination of both the Deep Residual Network (DRN) and Graph Convolutional Network (GCN).ResultsFurther, RF‐GCN is evaluated for performance by metrics such as accuracy, recall, and precision, with superior values of 91.6%, 96.5%, and 94.7%.ConclusionsRF‐GCN captures the nuanced relationships between different emotional states and improves recognition accuracy. The model is trained and evaluated on the dataset and reflects real‐world conditions.

A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection.

This paper presents a novel spatiotemporal transformer network that introduces several original components to detect actions in untrimmed videos. First, the multi-feature selective semantic attention model calculates the correlations between spatial and motion features to model spatiotemporal interactions between different action semantics properly. Second, the motion-aware network encodes the locations of action semantics in video frames utilizing the motion-aware 2D positional encoding algorithm. Such a motion-aware mechanism memorizes the dynamic spatiotemporal variations in action frames that current methods cannot exploit. Third, the sequence-based temporal attention model captures the heterogeneous temporal dependencies in action frames. In contrast to standard temporal attention used in natural language processing, primarily aimed at finding similarities between linguistic words, the proposed sequence-based temporal attention is designed to determine both the differences and similarities between video frames that jointly define the meaning of actions. The proposed approach outperforms the state-of-the-art solutions on four spatiotemporal action datasets: AVA 2.2, AVA 2.1, UCF101-24, and EPIC-Kitchens.

Real-Time CNN Training and Compression for Neural-Enhanced Adaptive Live Streaming.

We propose a real-time convolutional neural network (CNN) training and compression method for delivering high-quality live video even in a poor network environment. The server delivers a low-resolution video segment along with the corresponding CNN for super resolution (SR), after which the client applies the CNN to the segment in order to recover high-resolution video frames. To generate a trained CNN corresponding to a video segment in real-time, our method rapidly increases the training accuracy by promoting the overfitting property of the CNN while also using curriculum-based training. In addition, assuming that the pretrained CNN is already downloaded on the client side, we transfer only residual values between the updated and pretrained CNN parameters. These values can be quantized with low bits in real time while minimizing the amount of loss, as the distribution range is significantly narrower than that of the updated CNN. Quantitatively, our neural-enhanced adaptive live streaming pipeline (NEALS) achieves higher SR accuracy and a lower CNN compression loss rate within a constrained training time compared to the state-of-the-art CNN training and compression method. NEALS achieves 15 to 48% higher quality of the user experience compared to state-of-the-art neural-enhanced live streaming systems.

Combining optical flow and Swin Transformer for Space-Time video super-resolution

Space–time video super-resolution is a task that aims to interpolate low frame rate, low resolution videos to high frame rate, high resolution ones. While existing Transformer-based methods have achieved results comparable to convolutional neural networks-based methods, the computational cost of Transformer limits its performance with constrained computational resources. Moreover, Swin Transformer may fail to fully exploit the spatio-temporal information of video frames due to the limitation of window size, impeding its effectiveness in handling large motions. To address these limitations, we propose an end-to-end space–time video super-resolution architecture based on optical flow alignment and Swin Transformer. The alignment module is introduced to extract spatio-temporal information from adjacent frames without significantly increasing the computational burden. Additionally, we design a residual convolution layer to enhance the translational invariance of the features extracted by Swin Transformer and introduces additional nonlinear transformations. Experimental results demonstrate that our proposed method achieves superior performance on various benchmark datasets compared to state-of-the-art methods. In terms of Peak Signal-to-Noise Ratio, our method outperforms the state-of-the-art methods by at least 0.15 dB on Vimeo-Medium dataset.

Computer vision that can ‘see’ in the dark

Insufficient lighting environment has raised challenges for night shift workers’ safety monitoring. Thus, we have developed a computer vision-based algorithm recognizing 11 actions based on action recognition in dark (ARID) dataset. A hybrid model of integrating convolutional neural network (CNN) into YOLOv7 has been proposed. YOLOv7 is an algorithm designed for real-time object detection in image or video, for fast and accurate detection in applications such as autonomous vehicles and surveillance systems. In this work, video in dark environment has first been enhanced using CNN algorithm before feeding into YOLOv7 network for activity recognition. Adaptive gamma intensity correction (GIC) has been integrated to further improving the overall result. The proposed model has been evaluated over different enhancement modes. The proposed model is able to handle dark video frames with 74.95% Top-1 accuracy with fast processing speed of 93.99 ms/frame on a 4 GB RTX 3050 graphical processing unit (GPU) and 17.59 ms/frame on 16 GB Tesla T4 GPU. The base size of the proposed model is tiny, only 74.8 MB, but with 36.54 M of total parameters indicating that it has more capacity to learn more meaningful information with limited hardware resources.