Video Frames Research Articles

Eye Based Interactive Communication System

Eye-based communication systems, like the Blink-To-Speak language, are essential tools for individuals with motor neuron disorders, enabling them to articulate their needs and emotions. These systems leverage eye movements as a means of interaction, providing a vital communication channel for those with severe speech impairments. By interpreting specific eye gestures, users can effectively convey their thoughts, fostering a sense of independence and improving their quality of life. Many of the existing eye-tracking technologies tend to be complex and costly, limiting their accessibility in low-income regions. Using a mobile phone camera, the system captures real-time video frames to detect and track the user’s eyes, leveraging advanced facial landmark detection. Four primary eye movements—Left, Right, Up, and Blink—form the basis of the Blink-To-Live communication model, allowing users to communicate a range of more than 60 daily commands. By translating sequences of three eye gestures, the system generates readable sentences that are displayed on a screen and can be vocalized through synthesized speech. Unlike other sensor-dependent systems, Blink-To-Live is designed for affordability, flexibility, and ease of use, making it accessible without the need for specialized software or hardware. A prototype has been tested on a range of participants, showing positive results in terms of ease of use, flexibility, and cost-effectiveness

An image enhancement based method for improving rPPG extraction under low-light illumination

Remote photoplethysmography (rPPG) has gained significant attention as a non-invasive approach for measuring human vital signs from videos. However, the reliance on capturing the reflection of ambient lighting causes it to be susceptible to the adverse effects of low illumination levels, leading to deterioration in the signal quality. The preliminary study introduces a novel methodology for enhancing rPPG signal extraction in low-light conditions through the integration of an Image Enhancement Model (IEM) inspired by Retinex theory, which significantly improved signal quality by preprocessing video frames to better capture the subtle changes in facial blood flow. Recognizing the challenge posed by the generalization capacity over different deep-learning models and unseen examples from various datasets, this study further evaluated the efficacy of the IEM + rPPG extraction model across multiple datasets (UBFC-rPPG, BH-rPPG, MMPD-rPPG, and VIPL-HR) by integrating IEM with existing deep-learning based rPPG extraction models including DeepPhys, PhysNet, and PhysFormer and traditional POS rPPG extraction method. Our experiments demonstrate a consistent improvement for all the methods in heart rate estimation accuracy across all datasets, underscoring the IEM’s adaptability and effectiveness. The paper also explores the application of our method to traditional rPPG extraction techniques, further validating its potential for broader usage. Through comprehensive analysis, this research not only confirms the impact of lighting conditions on rPPG signal quality but also provides a robust solution for more reliable non-invasive vital sign monitoring in diverse environments.

Comparative Approach to De-Noising TEMPEST Video Frames

Analysis of unintended compromising emissions from Video Display Units (VDUs) is an important topic in research communities. This paper examines the feasibility of recovering the information displayed on the monitor from reconstructed video frames. The study holds particular significance for our understanding of security vulnerabilities associated with the electromagnetic radiation of digital displays. Considering the amount of noise that reconstructed TEMPEST video frames have, the work in this paper focuses on two different approaches to de-noising images for efficient optical character recognition. First, an Adaptive Wiener Filter (AWF) with adaptive window size implemented in the spatial domain was tested, and then a Convolutional Neural Network (CNN) with an encoder–decoder structure that follows both classical auto-encoder model architecture and U-Net architecture (auto-encoder with skip connections). These two techniques resulted in an improvement of more than two times on the Structural Similarity Index Metric (SSIM) for AWF and up to four times for the SSIM for the Deep Learning (DL) approach. In addition, to validate the results, the possibility of text recovery from processed noisy frames was studied using a state-of-the-art Tesseract Optical Character Recognition (OCR) engine. The present work aims to bring to attention the security importance of this topic and the non-negligible character of VDU information leakages.

SwinTExCo: Exemplar-based video colorization using Swin Transformer

Video colorization represents a compelling domain within the field of Computer Vision. The traditional approach in this field relies on Convolutional Neural Networks (CNNs) to extract features from each video frame and employs a recurrent network to learn information between video frames. While demonstrating considerable success in colorization, most traditional CNNs suffer from a limited receptive field size, capturing local information within a fixed-sized window. Consequently, they struggle to directly grasp long-range dependencies or pixel relationships that span large image or video frame areas. To address this limitation, recent advancements in the field have leveraged Vision Transformer (ViT) and their variants to enhance performance. This article introduces Swin Transformer Exemplar-based Video Colorization (SwinTExCo), an end-to-end model for the video colorization process that incorporates the Swin Transformer architecture as the backbone. The experimental results demonstrate that our proposed method outperforms many other state-of-the-art methods in both quantitative and qualitative metrics. The achievements of this research have significant implications for the domain of documentary and history video restoration, contributing to the broader goal of preserving cultural heritage and facilitating a deeper understanding of historical events through enhanced audiovisual materials.

Based on improved joint detection and tracking of UAV for multi-target detection of livestock

In agriculture, specifically livestock monitoring, drones' ability to track multiple targets is essential for advancing the field. However, limited computing resources and unpredictable drone movements often cause issues like ambiguous video frames, object obstructions, and size deviations. These inconsistencies reduce tracking accuracy, making traditional algorithms inadequate for handling drone footage. This study introduces an enhanced deep learning-based multi-target drone tracker framework that enables real-time processing. The proposed method combines object detection and tracking by leveraging consecutive frame pairs to extract and share features, enhancing computational efficiency. It employs diverse loss functions to address class and sample distribution imbalances and includes a composite deblurring module to enhance detection accuracy. Object association utilizes a dual regress bounding box technique, aiding in object identification verification and predictive motion. Live tracking is achieved by predicting object locations in subsequent frames, enabling real-time tracking. Evaluation against leading benchmarks shows that the system improves precision and speed, achieving a 4.3 % increase in Multi-Object Tracking Accuracy (MOTA) and a 7.7 % boost in F1 score.

IoT Video Delivery Optimization through Machine Learning-Based Frame Resolution Adjustment

Providing acceptable video quality in the Internet of Things (IoT) implementation poses a significant challenge, mainly when the application is performed on low-cost and low-power devices. This research focuses on developing a frame resolution adjustment system that maintains the frame rate value of video delivery in wireless IoT environments with resource-constrained devices. Consistent frame rates prevent motion lag and data loss, improving user experience. The system works by predicting the upcoming throughput values using machine learning methods to adjust the sensing parameter, which is the resolution of the video frame to be captured by camera nodes. Hence, the proposed system is equipped with a file size estimator to estimate the size of the next video frame and then adjust the resolution in accordance with the throughput prediction. In this research, we conducted extensive experiments to evaluate the accuracy of the file size estimator and the throughput prediction. The experiment generated a dataset to evaluate throughput prediction and file size estimator model. The evaluation results for the file size estimator showed a mean absolute percentage error (MAPE) of 6.73% in the experiment using 317 frames with video resolutions between 72p and 720p. Experiments were also conducted to compare several machine learning methods for predicting throughput values. Compared to long short-term memory (LSTM) and autoregressive integrated moving average (ARIMA), simple exponential smoothing (SES) outperforms the others with the lowest root mean squared error (RMSE) and mean absolute error (MAE) values. Building upon these findings, we implemented the frame resolution adjustment system using SES as the method for predicting the upcoming throughput values. Finally, we demonstrated that the proposed system can maintain the frame rate according to the threshold set by the system while the resolution is being maximized, thereby addressing the challenges of maintaining video quality in resource-constrained IoT environments.

A Video Dual-Domain Blind Watermarking Algorithm Based on Hadamard Transform

Addressing the compatibility challenges surrounding the robustness and reversibility of existing video watermarking techniques, this study introduces a novel video dual-domain blind watermarking algorithm leveraging the Hadamard transform. Specifically tailored for H.264 video copyright protection, the algorithm initially organizes video frames and identifies key frames for watermark embedding. Prior to embedding, the robust watermark undergoes coding preprocessing to optimize its integration. Subsequently, a 4×4 block is expanded based on the selected embedding position within the frame, followed by the application of the Hadamard transform to the enlarged block. The 1-bit robust watermark information is then embedded via the coefficient pair located in the first row of the Hadamard coefficient matrix corresponding to the expanded block. Additionally, a reversible watermark, designed to mitigate the distortions introduced during robust embedding, is generated and embedded into the remaining coefficients of the coefficient matrix using reversible embedding techniques. During watermark extraction, the dual-domain watermark can be retrieved exclusively through reversible extraction methodologies by analyzing the size relationship of coefficient pairs, eliminating the need for access to the original video data. To bolster the algorithm’s robustness, a majority-subordinate voting system is devised and implemented, effectively enhancing its resilience. Experimental findings demonstrate that, compared to similar approaches, this algorithm not only enhances the reversibility of video restoration but also exhibits superior robustness and meets the requirements for imperceptibility.

Enhancing Multi-Object Detection in Ultrasound Images Through Semi-Supervised Learning, Focal Loss and Relation of Frame

ObjectiveTo identify musculoskeletal anatomical structures in real time by using deep learning techniques. MethodsAn automated annotation system based on deep learning neural networks was designed to aid in the real-time identification of anatomical structures. Additionally, novel algorithms aimed at diminishing model training duration while enhancing accuracy were introduced. In this study, we proposed a semi-supervised learning (SSL) approach that substantially reduced annotation time. We also adopted the focal loss (FL) method to enhance the accuracy of challenging structures. Additionally, during the inference stage, we harnessed the temporal continuity of video frames, which involved leveraging information from preceding frames to facilitate recognition of structures in the current image. Training the model through a combination of SSL and FL yielded superior performance compared with supervised learning, while also substantially mitigating any expense linked to annotations. During inference, the incorporation of frame continuity helped to avoid discontinuity and bolster accuracy. ResultsForearm tissue detection was demonstrated by properly configuring the SSL approach, including FL and the filtering threshold. Comparable performance with supervised learning was achieved while only using 30% of the training data. The real-time experimental results also demonstrated that implementing relation of frame reduced the number of missing frames during inference and successfully increased the confidence scores of detected objects. ConclusionThis proposed system has the potential to aid medical professionals in efficiently and effectively diagnosing musculoskeletal disorders, ultimately leading to enhanced patient outcomes.

Snapshot computed tomographic microscopic imaging spectrometer and its video-level tracking of poisonous Microcystis aeruginosa cells in mixed algae

Conventional microscopic spectral imaging suffers from extended scanning times across wavelength or spatial dimension. To improve capabilities of dynamic microscopic spectral imaging, we developed a snapshot computed tomographic microscopic imaging spectrometer (CTMIS) based on the zeroth and first orders dispersive diffraction of a two-dimensional grating. Utilizing the CTMIS-UNET reconstruction algorithm, we can reconstruct a spectral cube (541x541x26) for each frame of micro spectral imaging video. Experimental results demonstrate a sub-4 μm spatial resolution achievable through a 20x objective lens and a spectral resolution better than 10 nm among 450–700 nm, while maintaining spectral cosine similarities exceeding 0.9989 when comparing reconstructed spectra with ground truth data. Spectral imaging videos of four species of algae and mixed algae were captured under 10 ms exposure time using the CTMIS system. Leveraging the self-developed UNET-SI26 algae recognition network, precise identification and tracking of four types of algae and poisonous microcysts aeruginosa in mixed algae were conducted. The pixel-level recognition accuracy exceeds 95 %, while the accuracy for counting the numbers of different types of cells surpasses 85 %, offering an efficient and accurate spectral imaging method for real-time monitoring and early warning of harmful algae at the cellular level.

Automatic Identification of Game Stuttering via Gameplay Videos Analysis

Modern video games are extremely complex software systems and, as such, they might suffer from several types of post-release issues. A particularly insidious issue is constituted by drops in the frame rate ( i.e. , stuttering events), which might have a negative impact on the user experience. Stuttering events are frequently documented in the million of hours of gameplay videos shared by players on platforms such as Twitch or YouTube. From the developers’ perspective, these videos represent a free source of documented “testing activities”. However, especially for popular games, the quantity and length of these videos make impractical their manual inspection. We introduce HASTE, an approach for the automatic detection of stuttering events in gameplay videos that can be exploited to generate candidate bug reports. HASTE firstly splits a given video into visually coherent slices, with the goal of filtering-out those that not representing actual gameplay ( e.g ., navigating the game settings). Then, it identifies the subset of pixels in the video frames which actually show the game in action excluding additional elements on screen such as the logo of the YouTube channel, on-screen chats etc. In this way, HASTE can exploit state-of-the-art image similarity metrics to identify candidate stuttering events, namely subsequent frames being almost identical in the pixels depicting the game. We evaluate the different steps behind HASTE on a total of 105 videos showing that it can correctly extract video slices with a 76% precision, and can correctly identify the slices related to gameplay with a recall and precision higher than 77%. Overall, HASTE achieves 71% recall and 89% precision for the identification of stuttering events in gameplay videos.

SimpleTrackV2: Rethinking the Timing Characteristics for Multi-Object Tracking.

Multi-object tracking tasks aim to assign unique trajectory codes to targets in video frames. Most detection-based tracking methods use Kalman filtering algorithms for trajectory prediction, directly utilizing associated target features for trajectory updates. However, this approach often fails, with camera jitter and transient target loss in real-world scenarios. This paper rethinks state prediction and fusion based on target temporal features to address these issues and proposes the SimpleTrackV2 algorithm, building on the previously designed SimpleTrack. Firstly, to address the poor prediction performance of linear motion models in complex scenes, we designed a target state prediction algorithm called LSTM-MP, based on long short-term memory (LSTM). This algorithm encodes the target's historical motion information using LSTM and decodes it with a multilayer perceptron (MLP) to achieve target state prediction. Secondly, to mitigate the effect of occlusion on target state saliency, we designed a spatiotemporal attention-based target appearance feature fusion (TSA-FF) target state fusion algorithm based on the attention mechanism. TSA-FF calculates adaptive fusion coefficients to enhance target state fusion, thereby improving the accuracy of subsequent data association. To demonstrate the effectiveness of the proposed method, we compared SimpleTrackV2 with the baseline model SimpleTrack on the MOT17 dataset. We also conducted ablation experiments on TSA-FF and LSTM-MP for SimpleTrackV2, exploring the optimal number of fusion frames and the impact of different loss functions on model performance. The experimental results show that SimpleTrackV2 handles camera jitter and target occlusion better, achieving improvements of 1.6%, 3.2%, and 6.1% in MOTA, IDF1, and HOTA, respectively, compared to the SimpleTrack algorithm.

Mask R-CNN and Centroid Tracking Algorithm to Process UAV Based Thermal-RGB Video for Drylot Cattle Heat Stress Monitoring

This study developed and evaluated an algorithm for processing thermal-RGB video feeds captured by an unmanned aerial vehicle (UAV) to automate heat stress monitoring in cattle housed in the drylots. The body surface temperature (BST) of individual cows was used as an indicator of heat stress. UAV data were collected using RGB and thermal infrared imagers, respectively, at 2 and 6.67 cm per pixel spatial resolution in Spring 2023 (dataset-1) and Summer 2024 (dataset-2). Study sites were two commercial drylots in Washington State. The custom algorithms were developed to: (1) detect and localize individual cows using a Mask R-CNN-based instance segmentation model combined with centroid tracking; and (2) extract BST by averaging the thermal-imagery pixels for each of the segmented cows. The algorithm showed higher detection accuracy with RGB images as input (F1 score: 0.89) compared to thermal (F1 score: 0.64). BST extraction with combined RGB and thermal imaging approach required corrections for alignment problems associated with differences in optics, imaging field of view, resolution, and lens properties. Consequently, thermal imaging-only approach was adopted for assessing real-time cow localization and BST estimation. Operating at one frame per second, algorithm successfully detected 72.4% and 81.65% of total cows in video frames from dataset-1 (38 s) and -2 (48 s), respectively. The mean absolute difference between algorithm output and ground truth (BSTGT) was 2.1 °C (dataset-1) and 3.3 °C (dataset-2), demonstrating satisfactory performance. With further refinements, this approach could be a viable tool for real-time heat stress monitoring in large-scale drylot production systems.

What physical examinations are observed during an in-person GP consultation? Automatic extraction using a text-based approach

ObjectivesTeleconsultation is anticipated to have a long-term role in primary care. However, conducting virtual physical examinations is a well-known limitation. To anticipate unmet needs general practitioners (GPs) and patients may experience during teleconsultation, this study aims to automatically identify physical examinations typically conducted during in-person GP consultation. Material and MethodsThis study utilizes 281 GP in-person consultations (de-identified transcripts & video recordings) within UK general practices, where 169 eligible ones were included in this study. We propose an automated text-based approach using regular expressions on keywords in GP-patient consultation dialogue (e.g., “roll up your sleeves”) to identify physical examinations (e.g. blood pressure measurement). This approach involves the construction of conceptual diagrams to visually inspect the relationship between keywords and physical examinations, syntax analysis to identify patterns between keywords and generate regular expressions, and the use of these regular expressions in consultation transcripts to detect potential instances of physical examinations, where matching video frames were subsequently retrieved. The performance of our automated text-based approach is compared to manual classification by 2 independent researchers using 5-fold cross-validation (precision, recall, and F1-score). ResultsAmong the 169 eligible GP in-person consultations, 133 (79%) required a physical examination, while the other 33 visits were for psychological reasons. Out of these 133 consultations, a total of 283 physical examinations were observed, with 21 instances conducted behind a curtain. We identified 42 distinct types of physical examinations from these 283 instances, grouped into 10 physical examination categories based on body areas and physical artefacts. The most frequent category of physical examinations is Vital Signs 26.80% (76/283). Overall, blood pressure measurement (also belonging to the Vital Signs category) is the most frequent physical examination at 59.2% (45/76). The comparison between manual classification and the regular expression model demonstrates an average precision of 88.3%, recall of 78.9%, and an F1-score of 83.3% from 5-fold cross-validation, providing significant insights into the frequency and types of physical examinations conducted during in-person GP consultations. ConclusionBy using regular expressions in consultation dialogues between GPs and patients, we can automatically identify physical examinations in GP consultations with a precision of 88.3%. Findings from this study, i.e. physical examinations during in-person GP consultations, provide insights into areas where GPs and patients may need support during teleconsultation.

Video Anomaly Detection via self-supervised and spatio-temporal proxy tasks learning

Video Anomaly Detection (VAD) aims to identify events in videos that deviate from typical patterns. Given the scarcity of anomalous samples, previous research has primarily focused on learning regular patterns from datasets exclusively containing normal behaviors, and treating deviations from these patterns as anomalies. However, most of these methods are constrained by coarse-grained modeling approaches that renders them incapable of learning highly-discriminative features, which are necessary to effectively distinguish between the subtle differences between normal and abnormal behaviors. To better capture these features, we propose an innovative method. Initially, pseudo-anomalous samples for appearance and motion are generated through geometric transformations (2D rotations) and the scrambling of video sequences. Subsequently, a dual-branch network featuring spatio-temporal decoupling is proposed, in which the spatial and temporal branches each handle a specific proxy task. These tasks are designed to distinguish between normal and pseudo-anomalous samples, involving operations such as predicting patch-based 2D rotation angles and classifying video frame triplets as total-anomaly, left-anomaly, right-anomaly, and non-anomaly. Our approach employs an end-to-end training methodology, without relying on pre-trained models (except for the object detector). Evaluations on the UCSD Ped2, Avenue, and ShanghaiTech datasets show that our method achieved AUC scores of 99.1%, 91.9%, and 81.1%, respectively, demonstrating its effectiveness. The code is publicly accessible at the following link: https://spatio-temporal-tasks.

Behavior recognition of cage-free multi-broilers based on spatiotemporal feature learning

Poultry behavior indicates their health, welfare, and production performance. Timely access to broilers' behavioral information can improve their welfare and reduce disease spread. Most behaviors require a period of observation before they can be accurately judged. However, the existing approaches for multi-object behavior recognition were mostly developed based on a single-frame image and ignored the temporal features in videos, which led to misrecognition. This study proposed an end-to-end method for recognizing multiple simultaneous behavioral events of cage-free broilers in videos by Broiler Behavior Recognition System (BBRS) based on spatiotemporal feature learning. The BBRS consisted of 3 main components: the improved YOLOv8s detector, the Bytetrack tracker, and the 3D-ResNet50-TSAM model. The basic network YOLOv8s was improved with MPDIoU to identify multiple broilers in the same frame of videos. The Bytetrack tracker was used to track each identified broiler and acquire its image sequence of 32 continuous frames as input for the 3D-ResNet50-TSAM model. To accurately recognize behavior of each tracked broiler, the 3D-ResNet50-TSAM model integrated a temporal-spatial attention module for learning the spatiotemporal features from its image sequence and enhancing inference ability in the case of its image sequence less than 32 continuous frames due to its tracker ID switching. Each component of BBRS was trained and tested with the rearing density of 7 to 8 birds/m2. The results demonstrated that the mAP@0.5 of the improved YOLOv8s detector was 99.50%. The Bytetrack tracker achieved a mean MOTA of 93.89% at different levels of occlusion. The Accuracy, Precision, Recall, and F1score of the 3D-ResNet50-TSAM model were 97.84, 97.72, 97.65, and 97.68%, respectively. The BBRS showed satisfactory inference ability with an Accuracy of 93.98% when 26 continuous frames of the tracked broiler were received by the 3D-ResNet50-TSAM model. This study provides an efficient tool for automatically and accurately recognizing behaviors of cage-free multi-broilers in videos. The code will be released on GitHub (https://github.com/CoderYLH/BBRS) as soon as the study is published.

Luminance decomposition and reconstruction for high dynamic range Video Quality Assessment

High dynamic range (HDR) video represents a wider range of brightness, detail and colour than standard dynamic range (SDR) video. However, SDR-based VQA (Video Quality Assessment) models struggle to capture HDR distortions. In addition, some of the existing methods designed for HDR video focus on emphasising the distortion of local areas of the video frame, ignoring the distortion of the video frame as a whole. Therefore, we propose a no reference VQA model based on luminance decomposition and recombination that provides excellent performance for HDR videos, called HDR-DRVQA. Specifically, HDR-DRVQA utilises a luminance decomposition strategy to decompose video frames into different regions for explicit extraction of perceptual features in different regions of the high dynamic range. We then further propose a residual aggregation module for recombining multi-region features to extract static spatial distortion representations and dynamic motion perception (captured by feature differences). Taking advantage of the Transformer network in remote dependency modelling, this information is fed into the Transformer network for interactive learning of motion perception and adaptively constructs a stream of spatial distortion information from shallow to deep layers during temporal aggregation. We validate that our model significantly outperforms SDR VQA and existing HDR VQA methods on the publicly available HDR databases.

Reconstruction of three-dimensional scenes based on video flow data

This work is dedicated to the application of modern algorithms for reconstructing spatial scenes from images to restore spatial information from video. The work is looking at a variety of modern methods, approaches, algorithms and trends in the field. The attention was paid to the sequence of development of approaches to the completion of the task. While researching the field and results related to three-dimensional reconstruction based on images and video streams, an algorithm was invented that allows constructing dense depth maps using information from all video frames. The idea is to use ready-made, commonly accepted, and tested solutions to solve two problems: COLMAP for visual odometry, and RAFT for computing optical flow. The algorithm shows quite accurate results and reconstructs the depth map in detail on arbitrary static scenes.

Video frame interpolation neural network for 3D tomography across different length scales

Three-dimensional (3D) tomography is a powerful investigative tool for many scientific domains, going from materials science, to engineering, to medicine. Many factors may limit the 3D resolution, often spatially anisotropic, compromising the precision of the information retrievable. A neural network, designed for video-frame interpolation, is employed to enhance tomographic images, achieving cubic-voxel resolution. The method is applied to distinct domains: the investigation of the morphology of printed graphene nanosheets networks, obtained via focused ion beam-scanning electron microscope (FIB-SEM), magnetic resonance imaging of the human brain, and X-ray computed tomography scans of the abdomen. The accuracy of the 3D tomographic maps can be quantified through computer-vision metrics, but most importantly with the precision on the physical quantities retrievable from the reconstructions, in the case of FIB-SEM the porosity, tortuosity, and effective diffusivity. This work showcases a versatile image-augmentation strategy for optimizing 3D tomography acquisition conditions, while preserving the information content.

A new method of intraoperative assessment of the dynamics of cortical blood flow using imaging photoplethysmography

Background. Intraoperative assessment of changes in cerebral blood flow is an important component of objective quality control of surgical treatment of cerebral artery aneurysms. Various techniques have been tried to solve this task, but they all have their drawbacks, which forces us to look for new ways of blood flow monitoring. We propose to use the technology of imaging photoplethysmography (IPPG) – a technically simple, contactless, safe and cheap optical method for assessing the perfusion of biological tissues.Aim. To demonstrate the possibility of using IPPG to assess the dynamics of cerebral blood flow parameters during aneurysm clipping surgery, as well as to identify early changes in blood supply to the cerebral cortex.Materials and methods. The study was carried out during six surgeries of clipping aneurysms of the anterior part of the Willis’s circle, both in the acute stage of rupture (n = 1) and in a planned manner (n = 5). The IPPG system, which is an LED illuminator in a single unit with a digital video camera, was located on a tripod 25 cm from the intervention zone. During each operation, two one-minute recordings of the illuminated surface of the cerebral cortex were performed: after dissection of the dura mater and before its suturing at the end of the main stage of the intervention. To improve the measurement accuracy, video frames of the studied area were recorded synchronously with the registration of an electrocardiogram. After recording, two IPPG parameters were calculated and compared: the amplitude of the pulsatile component and the pulse wave transit time. Thereafter, the obtained data were compared with the results of computed tomography. Statistical analysis was performed using pairwise comparison tests in the GraphPad Prism software package.Results. Clipping of cerebral vessel aneurysms are accompanied by significant changes in the parameters of cerebral blood flow. Analysis of the data for all patients revealed significant differences in IPPG parameters before and after surgery, namely, statistically significant increase in amplitude of the pulsatile component (n = 3) and decrease in pulse wave transit time (n = 5). The absence of significant changes in both parameters was found only in one patient who had mechanical damage in the cortex in the region of video recording.Conclusion. The IPPG system is capable to quantify changes in blood supply to the cortex during surgical treatment of cerebral artery aneurysms and to identify areas with either increased or decreased blood supply. In-depth studies are required to obtain additional markers of the postoperative state of cerebral blood flow.

Fishing event detection and species classification using computer vision and artificial intelligence for electronic monitoring

Fisheries regulations require detailed catch reporting on commercial fishing vessels. Vital components for the sustainable management of fish stocks include a robust estimate of the number of fish caught and the species composition. Catch recording is often done manually by human observers on fishing vessels. Human observers are costly, and consistent data streams can be subject to observer availability and the weather. On-vessel cameras (electronic monitoring, EM) are a growing alternative to human observers. However, on-land human auditors are required to review hundreds of hours of videos recorded during fishing trips that can last for weeks. In this paper, a framework is presented to automatically detect fish in EM videos, count the total fishing events, and classify the fish species. For this purpose, a deep learning and computer vision-based model is developed to efficiently detect fish and fishers onboard a vessel. Secondly, a vision-based tracking pipeline tracks the detected fish and counts the total fishing events in the videos. Thirdly, the extracted fishing events are classified through a deep learning-based fish species classifier, to provide the distribution of different fish species caught for a fishing trip. For our experiments, the datasets were prepared using the electronic monitoring data of multiple fishing trips of a fishing vessel. The videos were recorded on Australian longline vessels targeting tunas and billfish. For the fish detection task, video frames were extracted and labelled manually to provide a digital ground-truth. For the fish species classification task, hundreds of fish images of multiple species were cropped to provide a training dataset for the fish classifier. For the fish counting task, manual counts for the fishing events of individual fish species were generated for the test fishing trips. The developed fish and fisher detector achieves a mean Average Precision of 87.0 % for fish and 94.0 % for fishers on test video frames. The fishing event detection pipeline achieves an Average Precision of 81.0 % and an Average Recall of 74.5 % on test videos. The fish species classifier achieves an Accuracy (Top-1) of 91.11 % for the classification of cropped fish images and 89.05 % for the classification of extracted fishing events from the videos. Experimental results show that our proposed computer vision and artificial intelligence-based solution for video analysis has great potential to automate the auditing process from electronic monitoring footage and contribute to the sustainable management of fish stocks.