Video Frames Research Articles

A preliminary study of combustion flame by digital image processing and residue for fireworks flash powder

Fireworks play a vital role in festive celebrations and entertainment. These fireworks industries are administered by the Petroleum and Explosives Safety Organisation (PESO), Nagpur, Government of India. Even though PESO prescribed the standard composition for various fireworks products, concerns persist regarding the potential deviation from these standards, particularly in the pursuit of increased noise levels to attract consumers. This change in the burning and reactive properties of chemicals could potentially lead to accidents and environmental pollution. The current work investigated the combustion characteristics of two unknown market flash powder samples and a PESO standard sample. The samples were openly ignited in a controlled atmosphere, and video recordings of the resulting chemical flames were analyzed to determine fire area, flame intensity, and motion magnitude. The residues were also collected for EDAX and SEM analysis. Fire pixels from video frames are segmented to create a comprehensive database detailing flame area and motion magnitude under different circumstances. From the flame area profile, motion magnitude and SEM/EDAX analysis, it is found that PESO standard sample combustion is better than the unknown flash powder combustion.

EXCOR membrane motion analyzer (EMMA) to quantify and assess hemodynamic performance of the EXCOR pediatric heart assist device.

Pediatric heart failure is associated with high mortality rates and is a current clinical burden. There is only one FDA approved pediatric VAD, Berlin Heart EXCOR, for treatment. Thrombo-embolic complications are a significant clinical challenge, which can lead to devastating complications such as stroke and impair efficient EXCOR function. Currently, clinicians perform largely qualitative periodic assessment of EXCOR operation by observing the motion of a rapidly moving membrane, which can be prone to human error and can lead to missing out on crucial information. In this study, we design and implement a quantitative early warning system for accurate and quantitative assessment of the EXCOR membrane, named EXCOR Membrane Motion Analyzer (EMMA). Using a combination of image analysis, computer vision and custom designed algorithm, we perform rigorous frame by frame analysis of EXCOR membrane video data. We developed specialized metrics to identify relative smoothness between successive peaks, time between peaks and overall smoothness indicators to quantify and compare between multiple cases. Our results demonstrate that EMMA can successfully identify the motion and wrinkles on each video frame and quantify the smoothness and identify the phases of each cardiac cycle. Moreover, EMMA can obtain the smoothness of each frame and the temporal evolution of membrane smoothness across all image frames for the video sequence. EMMA allows for a fast, accurate, quantitative assessment to be completed and reduces user error. This enables EMMA to be used effectively as an early warning system to rapidly identify device abnormalities.

UAHOI: Uncertainty-aware robust interaction learning for HOI detection

This paper focuses on Human–Object Interaction (HOI) detection, addressing the challenge of identifying and understanding the interactions between humans and objects within a given image or video frame. Spearheaded by Detection Transformer (DETR), recent developments lead to significant improvements by replacing traditional region proposals by a set of learnable queries. However, despite the powerful representation capabilities provided by Transformers, existing Human–Object Interaction (HOI) detection methods still yield low confidence levels when dealing with complex interactions and are prone to overlooking interactive actions. To address these issues, we propose a novel approach UAHOI, Uncertainty-aware Robust Human–Object Interaction Learning that explicitly estimates prediction uncertainty during the training process to refine both detection and interaction predictions. Our model not only predicts the HOI triplets but also quantifies the uncertainty of these predictions. Specifically, we model this uncertainty through the variance of predictions and incorporate it into the optimization objective, allowing the model to adaptively adjust its confidence threshold based on prediction variance. This integration helps in mitigating the adverse effects of incorrect or ambiguous predictions that are common in traditional methods without any hand-designed components, serving as an automatic confidence threshold. Our method is flexible to existing HOI detection methods and demonstrates improved accuracy. We evaluate UAHOI on two standard benchmarks in the field: V-COCO and HICO-DET, which represent challenging scenarios for HOI detection. Through extensive experiments, we demonstrate that UAHOI achieves significant improvements over existing state-of-the-art methods, enhancing both the accuracy and robustness of HOI detection.

Hybridized deep learning goniometry for improved precision in Ehlers-Danlos Syndrome (EDS) evaluation

BackgroundGeneralized Joint Hyper-mobility (GJH) can aid in the diagnosis of Ehlers-Danlos Syndrome (EDS), a complex genetic connective tissue disorder with clinical features that can mimic other disease processes. Our study focuses on developing a unique image-based goniometry system, the HybridPoseNet, which utilizes a hybrid deep learning model.ObjectiveThe proposed model is designed to provide the most accurate joint angle measurements in EDS appraisals. Using a hybrid of CNNs and HyperLSTMs in the pose estimation module of HybridPoseNet offers superior generalization and time consistency properties, setting it apart from existing complex libraries.MethodologyHybridPoseNet integrates the spatial pattern recognition prowess of MobileNet-V2 with the sequential data processing capability of HyperLSTM units. The system captures the dynamic nature of joint motion by creating a model that learns from individual frames and the sequence of movements. The CNN module of HybridPoseNet was trained on a large and diverse data set before the fine-tuning of video data involving 50 individuals visiting the EDS clinic, focusing on joints that can hyperextend. HyperLSTMs have been incorporated in video frames to avoid any time breakage in joint angle estimation in consecutive frames. The model performance was evaluated using Spearman’s coefficient correlation versus manual goniometry measurements, as well as by the human labeling of joint position, the second validation step.OutcomePreliminary findings demonstrate HybridPoseNet achieving a remarkable correlation with manual Goniometric measurements: thumb (rho = 0.847), elbows (rho = 0.822), knees (rho = 0.839), and fifth fingers (rho = 0.896), indicating that the newest model is considerably better. The model manifested a consistent performance in all joint assessments, hence not requiring selecting a variety of pose-measuring libraries for every joint. The presentation of HybridPoseNet contributes to achieving a combined and normalized approach to reviewing the mobility of joints, which has an overall enhancement of approximately 20% in accuracy compared to the regular pose estimation libraries. This innovation is very valuable to the field of medical diagnostics of connective tissue diseases and a vast improvement to its understanding.

Video object detection via space–time feature aggregation and result reuse

AbstractWhen detecting the objects in videos, motion always leads to object deterioration, like blurring and occlusion, as well as the strange state of the object's shape and posture. Consequently, the detection of video frames will lead to a decline in accuracy by using the image object detection model. This paper proposes an online video object detection method based on the one‐stage detector YOLOx. First, the module for space–time feature aggregation is given, which uses the space–time information of past frames to enhance the feature quality of the current frame. Then, the module for result reuse is given, which incorporates the detection results of past frames to improve the detection stability of the current frame. By these two modules, the trade‐off between accuracy and speed of video object detection could be achieved. Experimental results on the ImageNet VID show the improvement of speed and accuracy of the proposed method.

A method to estimate prey densityfrom single-camera images: A case study with chinstrap penguins and Antarctic krill.

Estimating the densities of marine prey observed in animal-borne video loggers when encountered by foraging predators represents an important challenge for understanding predator-prey interactions in the marine environment. We used video images collected during the foraging trip of one chinstrap penguin (Pygoscelis antarcticus) from Cape Shirreff, Livingston Island, Antarctica to develop a novel approach for estimating the density of Antarctic krill (Euphausia superba) encountered during foraging activities. Using the open-source Video and Image Analytics for a Marine Environment (VIAME), we trained a neural network model to identify video frames containing krill. Our image classifier has an overall accuracy of 73%, with a positive predictive value of 83% for prediction of frames containing krill. We then developed a method to estimate the volume of water imaged, thus the density (N·m-3) of krill, in the 2-dimensional images. The method is based on the maximum range from the camera where krill remain visibly resolvable and assumes that mean krill length is known, and that the distribution of orientation angles of krill is uniform. From 1,932 images identified as containing krill, we manually identified a subset of 124 images from across the video record that contained resolvable and unresolvable krill necessary to estimate the resolvable range and imaged volume for the video sensor. Krill swarm density encountered by the penguins ranged from 2 to 307 krill·m-3 and mean density of krill was 48 krill·m-3 (sd = 61 krill·m-3). Mean krill biomass density was 25 g·m-3. Our frame-level image classifier model and krill density estimation method provide a new approach to efficiently process video-logger data and estimate krill density from 2D imagery, providing key information on prey aggregations that may affect predator foraging performance. The approach should be directly applicable to other marine predators feeding on aggregations of prey.

EMCFN: Edge-based Multi-scale Cross Fusion Network for video frame interpolation

Video frame interpolation (VFI) is used to synthesize one or more intermediate frames between two frames in a video sequence to improve the temporal resolution of the video. However, many methods still face challenges when dealing with complex scenes involving high-speed motion, occlusions, and other factors. To address these challenges, we propose an Edge-based Multi-scale Cross Fusion Network (EMCFN) for VFI. We integrate a feature enhancement module (FEM) based on edge information into the U-Net architecture, resulting in richer and more complete feature maps, while also enhancing the preservation of image structure and details. This contributes to generating more accurate and realistic interpolated frames. At the same time, we use a multi-scale cross fusion frame synthesis model (MCFM) composed of three GridNet branches to generate high-quality interpolation frames. We have conducted a series of experiments and the results show that our model exhibits satisfactory performance on different datasets compared with the state-of-the-art methods.

Augmented Reality Framework for Retrieving Information of Moving Objects on Construction Sites

The construction industry is undergoing a digital transformation, with the digital twin serving as a core system for project information. This digital twin provides an opportunity to utilize AR technology for real-time verification of on-site project information. Although many AR developments for construction sites have been attempted, they have been limited to accessing information on stationary components via Building Information Models. There have been no attempts to access information on dynamically changing resources, such as personnel and equipment. This paper addresses this gap by presenting an AR framework that enables site managers to verify real-time information on specific personnel or equipment. It introduces a matching algorithm for retrieving the necessary information from the digital twin. This algorithm is pivotal in identifying and retrieving the specific information needed from the vast dataset within the digital twin. The matching process integrates object detection and tracking algorithms applied to video frames from AR devices, along with GPS and IMU sensor data. Experimental results demonstrate the potential of this matching algorithm to streamline on-site management and reduce the effort required to interact with digital twin information. This paper highlights the transformative potential of AR and digital twin technologies in revolutionizing construction site operations.

Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features.

The cognitive state of a person can be categorized using the circumplex model of emotional states, a continuous model of two dimensions: arousal and valence. The purpose of this research is to select a machine learning model(s) to be integrated into a virtual reality (VR) system that runs cognitive remediation exercises for people with mental health disorders. As such, the prediction of emotional states is essential to customize treatments for those individuals. We exploit the Remote Collaborative and Affective Interactions (RECOLA) database to predict arousal and valence values using machine learning techniques. RECOLA includes audio, video, and physiological recordings of interactions between human participants. To allow learners to focus on the most relevant data, features are extracted from raw data. Such features can be predesigned, learned, or extracted implicitly using deep learners. Our previous work on video recordings focused on predesigned and learned visual features. In this paper, we extend our work onto deep visual features. Our deep visual features are extracted using the MobileNet-v2 convolutional neural network (CNN) that we previously trained on RECOLA's video frames of full/half faces. As the final purpose of our work is to integrate our solution into a practical VR application using head-mounted displays, we experimented with half faces as a proof of concept. The extracted deep features were then used to predict arousal and valence values via optimizable ensemble regression. We also fused the extracted visual features with the predesigned visual features and predicted arousal and valence values using the combined feature set. In an attempt to enhance our prediction performance, we further fused the predictions of the optimizable ensemble model with the predictions of the MobileNet-v2 model. After decision fusion, we achieved a root mean squared error (RMSE) of 0.1140, a Pearson's correlation coefficient (PCC) of 0.8000, and a concordance correlation coefficient (CCC) of 0.7868 on arousal predictions. We achieved an RMSE of 0.0790, a PCC of 0.7904, and a CCC of 0.7645 on valence predictions.

Efficient deep learning-based automated diagnosis from echocardiography with contrastive self-supervised learning

BackgroundAdvances in self-supervised learning (SSL) have enabled state-of-the-art automated medical image diagnosis from small, labeled datasets. This label efficiency is often desirable, given the difficulty of obtaining expert labels for medical image recognition tasks. However, most efforts toward SSL in medical imaging are not adapted to video-based modalities, such as echocardiography.MethodsWe developed a self-supervised contrastive learning approach, EchoCLR, for echocardiogram videos with the goal of learning strong representations for efficient fine-tuning on downstream cardiac disease diagnosis. EchoCLR pretraining involves (i) contrastive learning, where the model is trained to identify distinct videos of the same patient, and (ii) frame reordering, where the model is trained to predict the correct of video frames after being randomly shuffled.ResultsWhen fine-tuned on small portions of labeled data, EchoCLR pretraining significantly improves classification performance for left ventricular hypertrophy (LVH) and aortic stenosis (AS) over other transfer learning and SSL approaches across internal and external test sets. When fine-tuning on 10% of available training data (519 studies), an EchoCLR-pretrained model achieves 0.72 AUROC (95% CI: [0.69, 0.75]) on LVH classification, compared to 0.61 AUROC (95% CI: [0.57, 0.64]) with a standard transfer learning approach. Similarly, using 1% of available training data (53 studies), EchoCLR pretraining achieves 0.82 AUROC (95% CI: [0.79, 0.84]) on severe AS classification, compared to 0.61 AUROC (95% CI: [0.58, 0.65]) with transfer learning.ConclusionsEchoCLR is unique in its ability to learn representations of echocardiogram videos and demonstrates that SSL can enable label-efficient disease classification from small amounts of labeled data.

Collaborative Human-Computer Vision Operative Video Analysis Algorithm for Analyzing Surgical Fluency and Surgical Interruptions in Endonasal Endoscopic Pituitary Surgery: Cohort Study.

The endonasal endoscopic approach (EEA) is effective for pituitary adenoma resection. However, manual review of operative videos is time-consuming. The application of a computer vision (CV) algorithm could potentially reduce the time required for operative video review and facilitate the training of surgeons to overcome the learning curve of EEA. This study aimed to evaluate the performance of a CV-based video analysis system, based on OpenCV algorithm, to detect surgical interruptions and analyze surgical fluency in EEA. The accuracy of the CV-based video analysis was investigated, and the time required for operative video review using CV-based analysis was compared to that of manual review. The dominant color of each frame in the EEA video was determined using OpenCV. We developed an algorithm to identify events of surgical interruption if the alterations in the dominant color pixels reached certain thresholds. The thresholds were determined by training the current algorithm using EEA videos. The accuracy of the CV analysis was determined by manual review, and the time spent was reported. A total of 46 EEA operative videos were analyzed, with 93.6%, 95.1%, and 93.3% accuracies in the training, test 1, and test 2 data sets, respectively. Compared with manual review, CV-based analysis reduced the time required for operative video review by 86% (manual review: 166.8 and CV analysis: 22.6 minutes; P<.001). The application of a human-computer collaborative strategy increased the overall accuracy to 98.5%, with a 74% reduction in the review time (manual review: 166.8 and human-CV collaboration: 43.4 minutes; P<.001). Analysis of the different surgical phases showed that the sellar phase had the lowest frequency (nasal phase: 14.9, sphenoidal phase: 15.9, and sellar phase: 4.9 interruptions/10 minutes; P<.001) and duration (nasal phase: 67.4, sphenoidal phase: 77.9, and sellar phase: 31.1 seconds/10 minutes; P<.001) of surgical interruptions. A comparison of the early and late EEA videos showed that increased surgical experience was associated with a decreased number (early: 4.9 and late: 2.9 interruptions/10 minutes; P=.03) and duration (early: 41.1 and late: 19.8 seconds/10 minutes; P=.02) of surgical interruptions during the sellar phase. CV-based analysis had a 93% to 98% accuracy in detecting the number, frequency, and duration of surgical interruptions occurring during EEA. Moreover, CV-based analysis reduced the time required to analyze the surgical fluency in EEA videos compared to manual review. The application of CV can facilitate the training of surgeons to overcome the learning curve of endoscopic skull base surgery. ClinicalTrials.gov NCT06156020; https://clinicaltrials.gov/study/NCT06156020.

Rethink video retrieval representation for video captioning

Video captioning, a challenging task targeting the automatic generation of accurate and comprehensive descriptions based on video content, has witnessed substantial success recently driven by bridging video representations and textual semantics. Inspired by the nature of the video retrieval task, which learns visual features strongly related to text queries, we propose to take advantage of visual representation learning from the video retrieval framework to tackle video captioning tasks and construct adequate multi-grained cross-modal matching while extracting visual features. However, a simple direct application of recent video retrieval models fails to capture sufficient temporal details and the rich visual features of local patch tokens of video frames lack semantic information essential for captioning tasks. These deficiencies are primarily due to these models lack fine-grained interactions between video frames and offer only weak textual supervision over frame patch tokens. To increase the attention on temporal details, we propose a learnable token shift module, which flexibly captures subtle movements in local regions across the temporal sequence. Furthermore, we devise a Refineformer, which learns to integrate local video patch tokens strongly related to desired captions via a cross-attention mechanism. Extensive experiments on MSVD, MSR-VTT and VATEX demonstrate the favorable performance of our method. Code will be available at https://github.com/tiesanguaixia/IVRC.

Self-supervised multi-object tracking based on metric learning

The current paradigm of joint detection and tracking still requires a large amount of instance-level trajectory annotation, which incurs high annotation costs. Moreover, treating embedding training as a classification problem would lead to difficulties in model fitting. In this paper, we propose a new self-supervised multi-object tracking based on the real-time joint detection and embedding (JDE) framework, which we termed as self-supervised multi-object tracking (SS-MOT). In SS-MOT, the short-term temporal correlations between objects within and across adjacent video frames are both considered as self-supervised constraints, where the distances between different objects are enlarged while the distances between same object of adjacent frames are brought closer. In addition, short trajectories are formed by matching pairs of adjacent frames using a matching algorithm, and these matched pairs are treated as positive samples. The distances between positive samples are then minimized for futher the feature representation of the same object. Therefore, our method can be trained on videos without instance-level annotations. We apply our approach to state-of-the-art JDE models, such as FairMOT, Cstrack, and SiamMOT, and achieve comparable results to these supevised methods on the widely used MOT17 and MOT20 challenges.

Decoder-ROI based Versatile Video Coding for Multi-Object Tracking Vision Task

The video encoding standards High Efficiency Video Coding (HEVC) and, more recently, Versatile Video Coding (VVC) have introduced significant advancements in multimedia communication applications, such as video conferencing, broadcasting, and notably, E-learning. However, recent developments in artificial intelligence (AI) and big data have given rise to an urgent need for a specialized video encoding model designed specifically for image and video analysis applications using machine vision. In this paper, we propose a novel video encoding approach that effectively combines the ROI Coding algorithm and the VVC encoding model. The proposed method identifies regions of interest within video frames through fundamental and deep features. Based on this, we propose an adaptive compression method for each frame block, ensuring both the execution performance of machine learning applications and minimal data encoding requirements. To achieve new coding scheme without adding bitrate, New feature extraction approach are utilizing only decoded information (Decoder-ROI). The results demonstrate that the Decoder-ROI achieved significant compression rate improvement when compared to standard and relevant VCM schemes. Furthermore, ROI exploitation contributes to a 3.25\% reduction in encoding time compared to the baseline VVC encoding standard.

Semantic guidance incremental network for efficiency video super-resolution

AbstractIn video streaming, bandwidth constraints significantly affect client-side video quality. Addressing this, deep neural networks offer a promising avenue for implementing video super-resolution (VSR) at the user end, leveraging advancements in modern hardware, including mobile devices. The principal challenge in VSR is the computational intensity involved in processing temporal/spatial video data. Conventional methods, uniformly processing entire scenes, often result in inefficient resource allocation. This is evident in the over-processing of simpler regions and insufficient attention to complex regions, leading to edge artifacts in merged regions. Our innovative approach employs semantic segmentation and spatial frequency-based categorization to divide each video frame into regions of varying complexity: simple, medium, and complex. These are then processed through an efficient incremental model, optimizing computational resources. A key innovation is the sparse temporal/spatial feature transformation layer, which mitigates edge artifacts and ensures seamless integration of regional features, enhancing the naturalness of the super-resolution outcome. Experimental results demonstrate that our method significantly boosts VSR efficiency while maintaining effectiveness. This marks a notable advancement in streaming video technology, optimizing video quality with reduced computational demands. This approach, featuring semantic segmentation, spatial frequency analysis, and an incremental network structure, represents a substantial improvement over traditional VSR methodologies, addressing the core challenges of efficiency and quality in high-resolution video streaming.

A Proposed Text Encryption inside Video Using Harris Corner Detection and Salas20 Encryption Algorithm

يعد تشفير النص في الفيديو أداة حيوية في العديد من الصناعات حيث تكون حماية المعلومات الحساسة هي الأكثر أهمية. عادةً ما تواجه الخوارزميات وطرق الكشف الآلي صعوبة في تحديد موقع النص المخفي في إطارات الفيديو. تزيد هذه المقاومة من أمان المعلومات المخفية لأنه من الصعب على الأطراف غير المدعوة اكتشاف وجود نص مشفر. كما أنه يجمع بين المحتوى المكتوب والمرئي لتقديم كلا النوعين من المعلومات في وقت واحد وبسلاسة. في هذا البحث ، نقدم طريقة جديدة لتشفير النص داخل الفيديو باستخدام خوارزمية التشفير Salsa20 و Harris Corner Detection. الهدف من هذه الدراسة هو زيادة سرية وأمن المعلومات النصية المضمنة في محتوى الفيديو ، وهناك خطوتان أساسيتان في المنهجية المقترحة. لتحديد الزوايا أو النقاط المهمة داخل إطارات الفيديو ، تم استخدام طريقة Harris Corner Detection لأول مرة. تعمل هذه الزوايا كنقاط ميزة قوية يمكن استخدامها لدمج النص. تساعد طريقة Harris Corner Detection في تحديد الأماكن التي قد يتم إخفاء النص بها بدقة ، وذلك دون التقليل من جودة الصورة المرئية الإجمالية للفيديو ، وثانيًا ، يتم إنشاء مفتاح التشفير باستخدام تقنية التشفير Salsa20. Salsa20 هو تشفير دفق شائع يوفر تشفيرًا موثوقًا وتشغيلًا فعالاً. يتم تحويل النص المضمن إلى نص تشفير بواسطة XOR باستخدام مفتاح التشفير قبل تضمينه في الفيديو ، مما يضمن سريته وحمايته من الوصول غير المصرح به. تكشف مقارنة الطريقة المقترحة مع طرق تشفير النص الحالية عن عدد من المزايا. تدمج تقنية Harris Corner Detection النص في إطارات الفيديو بحيث يتم دمجها بشكل أقل وضوحًا ، مما يجعل من الصعب على الخصوم تحديد وجود المعلومات المخفية. يضمن استخدام Salsa20 أيضًا المستوى العالي من الأمان والحماية للنص المشفر ، والدفاع عنه ضد الهجمات المحتملة. يتم تقييم النظام المقترح باستخدام MSE و PSNR والارتباط و NPCR و UACI والإنتروبيا ، وتوفر مقاييس التقييم هذه نتائج ممتازة.

Enhancing Video Anomaly Detection Using a Transformer Spatiotemporal Attention Unsupervised Framework for Large Datasets

This work introduces an unsupervised framework for video anomaly detection, leveraging a hybrid deep learning model that combines a vision transformer (ViT) with a convolutional spatiotemporal relationship (STR) attention block. The proposed model addresses the challenges of anomaly detection in video surveillance by capturing both local and global relationships within video frames, a task that traditional convolutional neural networks (CNNs) often struggle with due to their localized field of view. We have utilized a pre-trained ViT as an encoder for feature extraction, which is then processed by the STR attention block to enhance the detection of spatiotemporal relationships among objects in videos. The novelty of this work is utilizing the ViT with the STR attention to detect video anomalies effectively in large and heterogeneous datasets, an important thing given the diverse environments and scenarios encountered in real-world surveillance. The framework was evaluated on three benchmark datasets, i.e., the UCSD-Ped2, CHUCK Avenue, and ShanghaiTech. This demonstrates the model’s superior performance in detecting anomalies compared to state-of-the-art methods, showcasing its potential to significantly enhance automated video surveillance systems by achieving area under the receiver operating characteristic curve (AUC ROC) values of 95.6, 86.8, and 82.1. To show the effectiveness of the proposed framework in detecting anomalies in extra-large datasets, we trained the model on a subset of the huge contemporary CHAD dataset that contains over 1 million frames, achieving AUC ROC values of 71.8 and 64.2 for CHAD-Cam 1 and CHAD-Cam 2, respectively, which outperforms the state-of-the-art techniques.

Phase-based Full-field Displacement Measurement with Phase Nonlinearity Weighting Process

The camera, as a non-contact sensor, has evolved into a robust tool for measuring the full-field vibration of complex structures. It offers distinct advantages over traditional contact sensors, including flexible positioning, simultaneous multi-point tracking, and high spatial resolution. Vibration imaging techniques have been developed to identify structural operational conditions through feature extraction methods. Phase-based vibration imaging techniques can visualize full-field operational shapes or vibrational features in a less-supervised manner. While phase-based optical flow with Gabor filters provides accurate displacement estimation, managing a large amount of image data can pose challenges in extracting full-field displacement efficiently. In this paper, we propose a compressive sensing approach for full-field phase-based optical flow and vibration imaging. Compressive sensing serves as an efficient sampling method that requires significantly fewer frames than traditional approaches, selecting frames at less than one third of the original video. Features are extracted from the displacement of the compressed video frames. Subsequently, the vibration features are transformed into images. The proposed methodology is validated through experiments conducted on an air compressor. Comparative results between vibration imaging from video with compressive sensing and that from the original video demonstrate that vibrational features can be extracted using only a few frames of the original video. Consequently, this approach can reduce processing time and data volume while preserving the performance of vibration feature extraction.

A deep-learning-based threshold-free method for automated analysis of rodent behavior in the forced swim test and tail suspension test

BackgroundThe forced swim test (FST) and tail suspension test (TST) are widely used to assess depressive-like behaviors in animals. Immobility time is used as an important parameter in both FST and TST. Traditional methods for analyzing FST and TST rely on manually setting the threshold for immobility, which is time-consuming and subjective. New methodWe proposed a threshold-free method for automated analysis of mice in these tests using a Dual-Stream Activity Analysis Network (DSAAN). Specifically, this network extracted spatial information of mice using a limited number of video frames and combined it with temporal information extracted from differential feature maps to determine the mouse’s state. To do so, we developed the Mouse FSTST dataset, which consisted of annotated video recordings of FST and TST. ResultsBy using DSAAN methods, we identify immobility states at accuracies of 92.51 % and 88.70 % for the TST and FST, respectively. The predicted immobility time from DSAAN is nicely correlated with a manual score, which indicates the reliability of the proposed method. Importantly, the DSAAN achieved over 80 % accuracy for both FST and TST by utilizing only 94 annotated images, suggesting that even a very limited training dataset can yield good performance in our model. Comparison with existing method(s)Compared with DBscorer and EthoVision XT, our method exhibits the highest Pearson correlation coefficient with manual annotation results on the Mouse FSTST dataset. ConclusionsWe established a powerful tool for analyzing depressive-like behavior independent of threshold, which is capable of freeing users from time-consuming manual analysis.

Estimating Surgical Urethral Length on Intraoperative Robot-Assisted Prostatectomy Images Using Artificial Intelligence Anatomy Recognition.

Objective: To construct a convolutional neural network (CNN) model that can recognize and delineate anatomic structures on intraoperative video frames of robot-assisted radical prostatectomy (RARP) and to use these annotations to predict the surgical urethral length (SUL). Background: Urethral dissection during RARP impacts patient urinary incontinence (UI) outcomes, and requires extensive training. Large differences exist between incontinence outcomes of different urologists and hospitals. Also, surgeon experience and education are critical toward optimal outcomes. Therefore, new approaches are warranted. SUL is associated with UI. Artificial intelligence (AI) surgical image segmentation using a CNN could automate SUL estimation and contribute toward future AI-assisted RARP and surgeon guidance. Methods: Eighty-eight intraoperative RARP videos between June 2009 and September 2014 were collected from a single center. Two hundred sixty-four frames were annotated according to prostate, urethra, ligated plexus, and catheter. Thirty annotated images from different RARP videos were used as a test data set. The dice (similarity) coefficient (DSC) and 95th percentile Hausdorff distance (Hd95) were used to determine model performance. SUL was calculated using the catheter as a reference. Results: The DSC of the best performing model were 0.735 and 0.755 for the catheter and urethra classes, respectively, with a Hd95 of 29.27 and 72.62, respectively. The model performed moderately on the ligated plexus and prostate. The predicted SUL showed a mean difference of 0.64 to 1.86 mm difference vs human annotators, but with significant deviation (standard deviation = 3.28-3.56). Conclusion: This study shows that an AI image segmentation model can predict vital structures during RARP urethral dissection with moderate to fair accuracy. SUL estimation derived from it showed large deviations and outliers when compared with human annotators, but with a small mean difference (<2 mm). This is a promising development for further research on AI-assisted RARP.