Video Frames Research Articles

Non-contact measurement of structural dynamic displacement based on improved edge detection and video interpolation

In structural health monitoring (SHM), smartphones are increasingly used for non-contact dynamic displacement measurement using computer vision. To overcome limitations in capturing high-frame-rate data, a non-contact measurement method of structural dynamic displacement based on an improved edge detection algorithm and video frame interpolation is proposed. Initially, a smartphone is employed to capture the structure’s vibration video and then generates intermediary frames between adjacent frames that match the structure’s motion state by using a video frame interpolation algorithm to obtain a video with a high frame rate. Subsequently, the improved edge detection algorithm is applied to measure the displacement within the interpolated video. In order to address the problem that traditional edge detection may have false edges leading to the degradation of measurement accuracy, a feature point tracking technique is proposed based on the traditional edge detection technique, which effectively excludes the interference of false edges in ROI and improves the accuracy of displacement monitoring. Experimental validation on a cantilever beam and steel frame bridge demonstrates the method’s viability and precision, highlighting its effectiveness in improving displacement measurement accuracy compared to traditional methods.

Fast and robust JND-guided video watermarking scheme in spatial domain

Generally speaking, those watermarking studies using the spatial domain tend to be fast but with limited robustness and imperceptibility while those performed in other transform domains are robust but have high computational cost. Watermarking applied to digital video has as one of the main challenges the large amount of computational power required due to the huge amount of information to be processed. In this paper we propose a watermarking algorithm for digital video that addresses this problem. To increase the speed, the watermark is embedded using a technique to modify the DCT coefficients directly in the spatial domain, in addition to carrying out this process considering the video scene as the basic unit and not the video frame. In terms of robustness, the watermark is modulated by a Just Noticeable Distortion (JND) scheme computed directly in the spatial domain guided by visual attention to increase the strength of the watermark to the maximum level but without this operation being perceivable by human eyes. Experimental results confirm that the proposed method achieves remarkable performance in terms of processing time, robustness and imperceptibility compared to previous studies.

Image colorization: A survey and dataset

Image colorization estimates RGB colors for grayscale images or video frames to improve their aesthetic and perceptual quality. Over the last decade, deep learning techniques for image colorization have significantly progressed, necessitating a systematic survey and benchmarking of these techniques. This article presents a comprehensive survey of recent state-of-the-art deep learning-based image colorization techniques, describing their fundamental block architectures, inputs, optimizers, loss functions, training protocols, training data, etc. It categorizes the existing colorization techniques into seven classes and discusses important factors governing their performance, such as benchmark datasets and evaluation metrics. We highlight the limitations of existing datasets and introduce a new dataset specific to colorization. We perform an extensive experimental evaluation of existing image colorization methods using both existing datasets and our proposed one. Finally, we discuss the limitations of existing methods and recommend possible solutions and future research directions for this rapidly evolving topic of deep image colorization. The dataset and codes for evaluation are publicly available at https://github.com/saeed-anwar/ColorSurvey.

AI-Driven Realtime Monitoring of Early Indicators for Ichthyophthirius multifiliis Infection of Rainbow Trout.

A novel video-based real-time system based on AI (artificial intelligence) was developed to detect clinical signs in fish exposed to pathogens. We selected a White Spot Disease model involving rainbow trout as the experimental animal and the parasitic ciliate Ichthyophthirius multifiliis as a pathogen. We compared two identical fish tank systems: one tank was infected by co-habitation, whereas the other tank was kept non-infected (sham infection). The two fish tanks were separately video monitored (full top and side view) during the course of infection, during which fish were removed whenever they developed clinical signs (direct visual inspection by the observer). Image analysis (object detection, classification and tracking) was used to track behavioural changes in fish (in every recorded video frame), focusing on movement patterns and spatial localisation. Initially, the two fish groups (infected and non-infected) exhibited similar behaviour and non-infected fish did not change behaviour during the 15 d observation period (from 5 d before infection until 10 dpi). At 4, 7, 8, 9 and 10 dpi some infected fish showed clinical signs (equilibrium disturbance, gasping and lethargy) and were removed from the experiment. Anorexia occurred from 5 dpi and a gradual progression of gasping behaviour was noted, whereas the frequency of fish flashing (rubbing/scratching against objects) was low. Equilibrium disturbances and the development of white spots in the skin appeared to be a much later (8-10 dpi at this temperature) indicator of infection. The video analysis showed a general distribution of non-infected fish in all parts of the fish tank during the entire experiment, whereas infected fish already at 4-5 dpi moved towards higher water currents in the top and bottom positions. This change of fish positioning within the tank appeared as a promising early indicator of infection. The study suggests that continuous monitoring of fish behaviour using AI can potentially optimise the timing of humane endpoints, indicate disease signs earlier and thereby improve animal welfare in both animal experimentation and in aquaculture settings.

Eye Based Interactive Communication System

Eye-based communication systems, like the Blink-To-Speak language, are essential tools for individuals with motor neuron disorders, enabling them to articulate their needs and emotions. These systems leverage eye movements as a means of interaction, providing a vital communication channel for those with severe speech impairments. By interpreting specific eye gestures, users can effectively convey their thoughts, fostering a sense of independence and improving their quality of life. Many of the existing eye-tracking technologies tend to be complex and costly, limiting their accessibility in low-income regions. Using a mobile phone camera, the system captures real-time video frames to detect and track the user’s eyes, leveraging advanced facial landmark detection. Four primary eye movements—Left, Right, Up, and Blink—form the basis of the Blink-To-Live communication model, allowing users to communicate a range of more than 60 daily commands. By translating sequences of three eye gestures, the system generates readable sentences that are displayed on a screen and can be vocalized through synthesized speech. Unlike other sensor-dependent systems, Blink-To-Live is designed for affordability, flexibility, and ease of use, making it accessible without the need for specialized software or hardware. A prototype has been tested on a range of participants, showing positive results in terms of ease of use, flexibility, and cost-effectiveness

An image enhancement based method for improving rPPG extraction under low-light illumination

Remote photoplethysmography (rPPG) has gained significant attention as a non-invasive approach for measuring human vital signs from videos. However, the reliance on capturing the reflection of ambient lighting causes it to be susceptible to the adverse effects of low illumination levels, leading to deterioration in the signal quality. The preliminary study introduces a novel methodology for enhancing rPPG signal extraction in low-light conditions through the integration of an Image Enhancement Model (IEM) inspired by Retinex theory, which significantly improved signal quality by preprocessing video frames to better capture the subtle changes in facial blood flow. Recognizing the challenge posed by the generalization capacity over different deep-learning models and unseen examples from various datasets, this study further evaluated the efficacy of the IEM + rPPG extraction model across multiple datasets (UBFC-rPPG, BH-rPPG, MMPD-rPPG, and VIPL-HR) by integrating IEM with existing deep-learning based rPPG extraction models including DeepPhys, PhysNet, and PhysFormer and traditional POS rPPG extraction method. Our experiments demonstrate a consistent improvement for all the methods in heart rate estimation accuracy across all datasets, underscoring the IEM’s adaptability and effectiveness. The paper also explores the application of our method to traditional rPPG extraction techniques, further validating its potential for broader usage. Through comprehensive analysis, this research not only confirms the impact of lighting conditions on rPPG signal quality but also provides a robust solution for more reliable non-invasive vital sign monitoring in diverse environments.

Comparative Approach to De-Noising TEMPEST Video Frames

Analysis of unintended compromising emissions from Video Display Units (VDUs) is an important topic in research communities. This paper examines the feasibility of recovering the information displayed on the monitor from reconstructed video frames. The study holds particular significance for our understanding of security vulnerabilities associated with the electromagnetic radiation of digital displays. Considering the amount of noise that reconstructed TEMPEST video frames have, the work in this paper focuses on two different approaches to de-noising images for efficient optical character recognition. First, an Adaptive Wiener Filter (AWF) with adaptive window size implemented in the spatial domain was tested, and then a Convolutional Neural Network (CNN) with an encoder–decoder structure that follows both classical auto-encoder model architecture and U-Net architecture (auto-encoder with skip connections). These two techniques resulted in an improvement of more than two times on the Structural Similarity Index Metric (SSIM) for AWF and up to four times for the SSIM for the Deep Learning (DL) approach. In addition, to validate the results, the possibility of text recovery from processed noisy frames was studied using a state-of-the-art Tesseract Optical Character Recognition (OCR) engine. The present work aims to bring to attention the security importance of this topic and the non-negligible character of VDU information leakages.

SwinTExCo: Exemplar-based video colorization using Swin Transformer

Video colorization represents a compelling domain within the field of Computer Vision. The traditional approach in this field relies on Convolutional Neural Networks (CNNs) to extract features from each video frame and employs a recurrent network to learn information between video frames. While demonstrating considerable success in colorization, most traditional CNNs suffer from a limited receptive field size, capturing local information within a fixed-sized window. Consequently, they struggle to directly grasp long-range dependencies or pixel relationships that span large image or video frame areas. To address this limitation, recent advancements in the field have leveraged Vision Transformer (ViT) and their variants to enhance performance. This article introduces Swin Transformer Exemplar-based Video Colorization (SwinTExCo), an end-to-end model for the video colorization process that incorporates the Swin Transformer architecture as the backbone. The experimental results demonstrate that our proposed method outperforms many other state-of-the-art methods in both quantitative and qualitative metrics. The achievements of this research have significant implications for the domain of documentary and history video restoration, contributing to the broader goal of preserving cultural heritage and facilitating a deeper understanding of historical events through enhanced audiovisual materials.

Based on improved joint detection and tracking of UAV for multi-target detection of livestock

In agriculture, specifically livestock monitoring, drones' ability to track multiple targets is essential for advancing the field. However, limited computing resources and unpredictable drone movements often cause issues like ambiguous video frames, object obstructions, and size deviations. These inconsistencies reduce tracking accuracy, making traditional algorithms inadequate for handling drone footage. This study introduces an enhanced deep learning-based multi-target drone tracker framework that enables real-time processing. The proposed method combines object detection and tracking by leveraging consecutive frame pairs to extract and share features, enhancing computational efficiency. It employs diverse loss functions to address class and sample distribution imbalances and includes a composite deblurring module to enhance detection accuracy. Object association utilizes a dual regress bounding box technique, aiding in object identification verification and predictive motion. Live tracking is achieved by predicting object locations in subsequent frames, enabling real-time tracking. Evaluation against leading benchmarks shows that the system improves precision and speed, achieving a 4.3 % increase in Multi-Object Tracking Accuracy (MOTA) and a 7.7 % boost in F1 score.

IoT Video Delivery Optimization through Machine Learning-Based Frame Resolution Adjustment

Providing acceptable video quality in the Internet of Things (IoT) implementation poses a significant challenge, mainly when the application is performed on low-cost and low-power devices. This research focuses on developing a frame resolution adjustment system that maintains the frame rate value of video delivery in wireless IoT environments with resource-constrained devices. Consistent frame rates prevent motion lag and data loss, improving user experience. The system works by predicting the upcoming throughput values using machine learning methods to adjust the sensing parameter, which is the resolution of the video frame to be captured by camera nodes. Hence, the proposed system is equipped with a file size estimator to estimate the size of the next video frame and then adjust the resolution in accordance with the throughput prediction. In this research, we conducted extensive experiments to evaluate the accuracy of the file size estimator and the throughput prediction. The experiment generated a dataset to evaluate throughput prediction and file size estimator model. The evaluation results for the file size estimator showed a mean absolute percentage error (MAPE) of 6.73% in the experiment using 317 frames with video resolutions between 72p and 720p. Experiments were also conducted to compare several machine learning methods for predicting throughput values. Compared to long short-term memory (LSTM) and autoregressive integrated moving average (ARIMA), simple exponential smoothing (SES) outperforms the others with the lowest root mean squared error (RMSE) and mean absolute error (MAE) values. Building upon these findings, we implemented the frame resolution adjustment system using SES as the method for predicting the upcoming throughput values. Finally, we demonstrated that the proposed system can maintain the frame rate according to the threshold set by the system while the resolution is being maximized, thereby addressing the challenges of maintaining video quality in resource-constrained IoT environments.

A Video Dual-Domain Blind Watermarking Algorithm Based on Hadamard Transform

Addressing the compatibility challenges surrounding the robustness and reversibility of existing video watermarking techniques, this study introduces a novel video dual-domain blind watermarking algorithm leveraging the Hadamard transform. Specifically tailored for H.264 video copyright protection, the algorithm initially organizes video frames and identifies key frames for watermark embedding. Prior to embedding, the robust watermark undergoes coding preprocessing to optimize its integration. Subsequently, a 4×4 block is expanded based on the selected embedding position within the frame, followed by the application of the Hadamard transform to the enlarged block. The 1-bit robust watermark information is then embedded via the coefficient pair located in the first row of the Hadamard coefficient matrix corresponding to the expanded block. Additionally, a reversible watermark, designed to mitigate the distortions introduced during robust embedding, is generated and embedded into the remaining coefficients of the coefficient matrix using reversible embedding techniques. During watermark extraction, the dual-domain watermark can be retrieved exclusively through reversible extraction methodologies by analyzing the size relationship of coefficient pairs, eliminating the need for access to the original video data. To bolster the algorithm’s robustness, a majority-subordinate voting system is devised and implemented, effectively enhancing its resilience. Experimental findings demonstrate that, compared to similar approaches, this algorithm not only enhances the reversibility of video restoration but also exhibits superior robustness and meets the requirements for imperceptibility.

Enhancing Multi-Object Detection in Ultrasound Images Through Semi-Supervised Learning, Focal Loss and Relation of Frame

ObjectiveTo identify musculoskeletal anatomical structures in real time by using deep learning techniques. MethodsAn automated annotation system based on deep learning neural networks was designed to aid in the real-time identification of anatomical structures. Additionally, novel algorithms aimed at diminishing model training duration while enhancing accuracy were introduced. In this study, we proposed a semi-supervised learning (SSL) approach that substantially reduced annotation time. We also adopted the focal loss (FL) method to enhance the accuracy of challenging structures. Additionally, during the inference stage, we harnessed the temporal continuity of video frames, which involved leveraging information from preceding frames to facilitate recognition of structures in the current image. Training the model through a combination of SSL and FL yielded superior performance compared with supervised learning, while also substantially mitigating any expense linked to annotations. During inference, the incorporation of frame continuity helped to avoid discontinuity and bolster accuracy. ResultsForearm tissue detection was demonstrated by properly configuring the SSL approach, including FL and the filtering threshold. Comparable performance with supervised learning was achieved while only using 30% of the training data. The real-time experimental results also demonstrated that implementing relation of frame reduced the number of missing frames during inference and successfully increased the confidence scores of detected objects. ConclusionThis proposed system has the potential to aid medical professionals in efficiently and effectively diagnosing musculoskeletal disorders, ultimately leading to enhanced patient outcomes.

Snapshot computed tomographic microscopic imaging spectrometer and its video-level tracking of poisonous Microcystis aeruginosa cells in mixed algae

Conventional microscopic spectral imaging suffers from extended scanning times across wavelength or spatial dimension. To improve capabilities of dynamic microscopic spectral imaging, we developed a snapshot computed tomographic microscopic imaging spectrometer (CTMIS) based on the zeroth and first orders dispersive diffraction of a two-dimensional grating. Utilizing the CTMIS-UNET reconstruction algorithm, we can reconstruct a spectral cube (541x541x26) for each frame of micro spectral imaging video. Experimental results demonstrate a sub-4 μm spatial resolution achievable through a 20x objective lens and a spectral resolution better than 10 nm among 450–700 nm, while maintaining spectral cosine similarities exceeding 0.9989 when comparing reconstructed spectra with ground truth data. Spectral imaging videos of four species of algae and mixed algae were captured under 10 ms exposure time using the CTMIS system. Leveraging the self-developed UNET-SI26 algae recognition network, precise identification and tracking of four types of algae and poisonous microcysts aeruginosa in mixed algae were conducted. The pixel-level recognition accuracy exceeds 95 %, while the accuracy for counting the numbers of different types of cells surpasses 85 %, offering an efficient and accurate spectral imaging method for real-time monitoring and early warning of harmful algae at the cellular level.

Automatic Identification of Game Stuttering via Gameplay Videos Analysis

Modern video games are extremely complex software systems and, as such, they might suffer from several types of post-release issues. A particularly insidious issue is constituted by drops in the frame rate ( i.e. , stuttering events), which might have a negative impact on the user experience. Stuttering events are frequently documented in the million of hours of gameplay videos shared by players on platforms such as Twitch or YouTube. From the developers’ perspective, these videos represent a free source of documented “testing activities”. However, especially for popular games, the quantity and length of these videos make impractical their manual inspection. We introduce HASTE, an approach for the automatic detection of stuttering events in gameplay videos that can be exploited to generate candidate bug reports. HASTE firstly splits a given video into visually coherent slices, with the goal of filtering-out those that not representing actual gameplay ( e.g ., navigating the game settings). Then, it identifies the subset of pixels in the video frames which actually show the game in action excluding additional elements on screen such as the logo of the YouTube channel, on-screen chats etc. In this way, HASTE can exploit state-of-the-art image similarity metrics to identify candidate stuttering events, namely subsequent frames being almost identical in the pixels depicting the game. We evaluate the different steps behind HASTE on a total of 105 videos showing that it can correctly extract video slices with a 76% precision, and can correctly identify the slices related to gameplay with a recall and precision higher than 77%. Overall, HASTE achieves 71% recall and 89% precision for the identification of stuttering events in gameplay videos.

SimpleTrackV2: Rethinking the Timing Characteristics for Multi-Object Tracking.

Multi-object tracking tasks aim to assign unique trajectory codes to targets in video frames. Most detection-based tracking methods use Kalman filtering algorithms for trajectory prediction, directly utilizing associated target features for trajectory updates. However, this approach often fails, with camera jitter and transient target loss in real-world scenarios. This paper rethinks state prediction and fusion based on target temporal features to address these issues and proposes the SimpleTrackV2 algorithm, building on the previously designed SimpleTrack. Firstly, to address the poor prediction performance of linear motion models in complex scenes, we designed a target state prediction algorithm called LSTM-MP, based on long short-term memory (LSTM). This algorithm encodes the target's historical motion information using LSTM and decodes it with a multilayer perceptron (MLP) to achieve target state prediction. Secondly, to mitigate the effect of occlusion on target state saliency, we designed a spatiotemporal attention-based target appearance feature fusion (TSA-FF) target state fusion algorithm based on the attention mechanism. TSA-FF calculates adaptive fusion coefficients to enhance target state fusion, thereby improving the accuracy of subsequent data association. To demonstrate the effectiveness of the proposed method, we compared SimpleTrackV2 with the baseline model SimpleTrack on the MOT17 dataset. We also conducted ablation experiments on TSA-FF and LSTM-MP for SimpleTrackV2, exploring the optimal number of fusion frames and the impact of different loss functions on model performance. The experimental results show that SimpleTrackV2 handles camera jitter and target occlusion better, achieving improvements of 1.6%, 3.2%, and 6.1% in MOTA, IDF1, and HOTA, respectively, compared to the SimpleTrack algorithm.

Mask R-CNN and Centroid Tracking Algorithm to Process UAV Based Thermal-RGB Video for Drylot Cattle Heat Stress Monitoring

This study developed and evaluated an algorithm for processing thermal-RGB video feeds captured by an unmanned aerial vehicle (UAV) to automate heat stress monitoring in cattle housed in the drylots. The body surface temperature (BST) of individual cows was used as an indicator of heat stress. UAV data were collected using RGB and thermal infrared imagers, respectively, at 2 and 6.67 cm per pixel spatial resolution in Spring 2023 (dataset-1) and Summer 2024 (dataset-2). Study sites were two commercial drylots in Washington State. The custom algorithms were developed to: (1) detect and localize individual cows using a Mask R-CNN-based instance segmentation model combined with centroid tracking; and (2) extract BST by averaging the thermal-imagery pixels for each of the segmented cows. The algorithm showed higher detection accuracy with RGB images as input (F1 score: 0.89) compared to thermal (F1 score: 0.64). BST extraction with combined RGB and thermal imaging approach required corrections for alignment problems associated with differences in optics, imaging field of view, resolution, and lens properties. Consequently, thermal imaging-only approach was adopted for assessing real-time cow localization and BST estimation. Operating at one frame per second, algorithm successfully detected 72.4% and 81.65% of total cows in video frames from dataset-1 (38 s) and -2 (48 s), respectively. The mean absolute difference between algorithm output and ground truth (BSTGT) was 2.1 °C (dataset-1) and 3.3 °C (dataset-2), demonstrating satisfactory performance. With further refinements, this approach could be a viable tool for real-time heat stress monitoring in large-scale drylot production systems.

What physical examinations are observed during an in-person GP consultation? Automatic extraction using a text-based approach

ObjectivesTeleconsultation is anticipated to have a long-term role in primary care. However, conducting virtual physical examinations is a well-known limitation. To anticipate unmet needs general practitioners (GPs) and patients may experience during teleconsultation, this study aims to automatically identify physical examinations typically conducted during in-person GP consultation. Material and MethodsThis study utilizes 281 GP in-person consultations (de-identified transcripts & video recordings) within UK general practices, where 169 eligible ones were included in this study. We propose an automated text-based approach using regular expressions on keywords in GP-patient consultation dialogue (e.g., “roll up your sleeves”) to identify physical examinations (e.g. blood pressure measurement). This approach involves the construction of conceptual diagrams to visually inspect the relationship between keywords and physical examinations, syntax analysis to identify patterns between keywords and generate regular expressions, and the use of these regular expressions in consultation transcripts to detect potential instances of physical examinations, where matching video frames were subsequently retrieved. The performance of our automated text-based approach is compared to manual classification by 2 independent researchers using 5-fold cross-validation (precision, recall, and F1-score). ResultsAmong the 169 eligible GP in-person consultations, 133 (79%) required a physical examination, while the other 33 visits were for psychological reasons. Out of these 133 consultations, a total of 283 physical examinations were observed, with 21 instances conducted behind a curtain. We identified 42 distinct types of physical examinations from these 283 instances, grouped into 10 physical examination categories based on body areas and physical artefacts. The most frequent category of physical examinations is Vital Signs 26.80% (76/283). Overall, blood pressure measurement (also belonging to the Vital Signs category) is the most frequent physical examination at 59.2% (45/76). The comparison between manual classification and the regular expression model demonstrates an average precision of 88.3%, recall of 78.9%, and an F1-score of 83.3% from 5-fold cross-validation, providing significant insights into the frequency and types of physical examinations conducted during in-person GP consultations. ConclusionBy using regular expressions in consultation dialogues between GPs and patients, we can automatically identify physical examinations in GP consultations with a precision of 88.3%. Findings from this study, i.e. physical examinations during in-person GP consultations, provide insights into areas where GPs and patients may need support during teleconsultation.

Video Anomaly Detection via self-supervised and spatio-temporal proxy tasks learning

Video Anomaly Detection (VAD) aims to identify events in videos that deviate from typical patterns. Given the scarcity of anomalous samples, previous research has primarily focused on learning regular patterns from datasets exclusively containing normal behaviors, and treating deviations from these patterns as anomalies. However, most of these methods are constrained by coarse-grained modeling approaches that renders them incapable of learning highly-discriminative features, which are necessary to effectively distinguish between the subtle differences between normal and abnormal behaviors. To better capture these features, we propose an innovative method. Initially, pseudo-anomalous samples for appearance and motion are generated through geometric transformations (2D rotations) and the scrambling of video sequences. Subsequently, a dual-branch network featuring spatio-temporal decoupling is proposed, in which the spatial and temporal branches each handle a specific proxy task. These tasks are designed to distinguish between normal and pseudo-anomalous samples, involving operations such as predicting patch-based 2D rotation angles and classifying video frame triplets as total-anomaly, left-anomaly, right-anomaly, and non-anomaly. Our approach employs an end-to-end training methodology, without relying on pre-trained models (except for the object detector). Evaluations on the UCSD Ped2, Avenue, and ShanghaiTech datasets show that our method achieved AUC scores of 99.1%, 91.9%, and 81.1%, respectively, demonstrating its effectiveness. The code is publicly accessible at the following link: https://spatio-temporal-tasks.

Behavior recognition of cage-free multi-broilers based on spatiotemporal feature learning

Poultry behavior indicates their health, welfare, and production performance. Timely access to broilers' behavioral information can improve their welfare and reduce disease spread. Most behaviors require a period of observation before they can be accurately judged. However, the existing approaches for multi-object behavior recognition were mostly developed based on a single-frame image and ignored the temporal features in videos, which led to misrecognition. This study proposed an end-to-end method for recognizing multiple simultaneous behavioral events of cage-free broilers in videos by Broiler Behavior Recognition System (BBRS) based on spatiotemporal feature learning. The BBRS consisted of 3 main components: the improved YOLOv8s detector, the Bytetrack tracker, and the 3D-ResNet50-TSAM model. The basic network YOLOv8s was improved with MPDIoU to identify multiple broilers in the same frame of videos. The Bytetrack tracker was used to track each identified broiler and acquire its image sequence of 32 continuous frames as input for the 3D-ResNet50-TSAM model. To accurately recognize behavior of each tracked broiler, the 3D-ResNet50-TSAM model integrated a temporal-spatial attention module for learning the spatiotemporal features from its image sequence and enhancing inference ability in the case of its image sequence less than 32 continuous frames due to its tracker ID switching. Each component of BBRS was trained and tested with the rearing density of 7 to 8 birds/m2. The results demonstrated that the mAP@0.5 of the improved YOLOv8s detector was 99.50%. The Bytetrack tracker achieved a mean MOTA of 93.89% at different levels of occlusion. The Accuracy, Precision, Recall, and F1score of the 3D-ResNet50-TSAM model were 97.84, 97.72, 97.65, and 97.68%, respectively. The BBRS showed satisfactory inference ability with an Accuracy of 93.98% when 26 continuous frames of the tracked broiler were received by the 3D-ResNet50-TSAM model. This study provides an efficient tool for automatically and accurately recognizing behaviors of cage-free multi-broilers in videos. The code will be released on GitHub (https://github.com/CoderYLH/BBRS) as soon as the study is published.

Reconstruction of three-dimensional scenes based on video flow data

This work is dedicated to the application of modern algorithms for reconstructing spatial scenes from images to restore spatial information from video. The work is looking at a variety of modern methods, approaches, algorithms and trends in the field. The attention was paid to the sequence of development of approaches to the completion of the task. While researching the field and results related to three-dimensional reconstruction based on images and video streams, an algorithm was invented that allows constructing dense depth maps using information from all video frames. The idea is to use ready-made, commonly accepted, and tested solutions to solve two problems: COLMAP for visual odometry, and RAFT for computing optical flow. The algorithm shows quite accurate results and reconstructs the depth map in detail on arbitrary static scenes.