Raw Video Data Research Articles

Demixing fluorescence time traces transmitted by multimode fibers

Optical methods based on thin multimode fibers (MMFs) are promising tools for measuring neuronal activity in deep brain regions of freely moving mice thanks to their small diameter. However, current methods are limited: while fiber photometry provides only ensemble activity, imaging techniques using of long multimode fibers are very sensitive to bending and have not been applied to unrestrained rodents yet. Here, we demonstrate the fundamentals of a new approach using a short MMF coupled to a miniscope. In proof-of-principle in vitro experiments, we disentangled spatio-temporal fluorescence signals from multiple fluorescent sources transmitted by a thin (200 µm) and short (8 mm) MMF, using a general unconstrained non-negative matrix factorization algorithm directly on the raw video data. Furthermore, we show that low-cost open-source miniscopes have sufficient sensitivity to image the same fluorescence patterns seen in our proof-of-principle experiment, suggesting a new avenue for novel minimally invasive deep brain studies using multimode fibers in freely behaving mice.

Astrocytic Calcium Signaling Toolkit (astroCaST): efficient analysis of dynamic astrocytic calcium events.

The Astrocytic Calcium Signaling Toolkit (astroCaST) is a novel solution to a longstanding challenge in neuroscience research: the specialized analysis of astrocytic calcium events within fluorescence time-series imaging. Distinct from existing neuron-centric tools, astroCaST is adept at detecting and clustering astrocytic calcium events based on their unique spatiotemporal characteristics, thus filling a gap in astrocytic research methodologies. This toolkit not only facilitates the detection of such events but also extends its utility to provide comprehensive end-to-end analysis. This feature is absent in most tools targeting astrocytic activity. AstroCaST's development was motivated by the critical need for dedicated software that supports researchers in transitioning from raw video data to insightful experimental conclusions, efficiently managing large-scale datasets without compromising computational speed. It offers a user-friendly interface that caters to both novice and expert users, incorporating both a graphical user interface (GUI) for detailed explorations and a command-line interface (CLI) for extensive analyses. Expected outcomes from utilizing astroCaST include the ability to process and analyze a significantly larger volume of data. This enables a more profound and comprehensive analysis than previously possible, addressing the demands of large-scale astrocytic studies. In summary, astroCaST aims to advance astrocytic calcium imaging analysis, offering a tailored, efficient, and comprehensive toolset that enhances our understanding of astrocytic functions and their implications in neuroscience.

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Current methods for assessing individual well-being in team collaboration at the workplace often rely on manually collected surveys. This limits continuous real-world data collection and proactive measures to improve team member workplace satisfaction. We propose a method to automatically derive social signals related to individual well-being in team collaboration from raw audio and video data collected in teamwork contexts. The goal was to develop computational methods and measurements to facilitate the mirroring of individuals’ well-being to themselves. We focus on how speech behavior is perceived by team members to improve their well-being. Our main contribution is the assembly of an integrated toolchain to perform multi-modal extraction of robust speech features in noisy field settings and to explore which features are predictors of self-reported satisfaction scores. We applied the toolchain to a case study, where we collected videos of 20 teams with 56 participants collaborating over a four-day period in a team project in an educational environment. Our audiovisual speaker diarization extracted individual speech features from a noisy environment. As the dependent variable, team members filled out a daily PERMA (positive emotion, engagement, relationships, meaning, and accomplishment) survey. These well-being scores were predicted using speech features extracted from the videos using machine learning. The results suggest that the proposed toolchain was able to automatically predict individual well-being in teams, leading to better teamwork and happier team members.

Motion Vector-Based Self-Attention for Real-Time Human Activity Recognition in Compressed Videos: The MVViT Approach

Herein, a novel methodology is proposed for real-time recognition of human activity in a compressed domain of videos based on motion vectors and self-attention mechanism using vision transformers, and it is termed as motion vectors and vision transformers (MVViT). The videos in MPEG-4 and H.264 compression formats are considered for this study. Any video source without any prior setup could be considered by adopting the proposed method to the corresponding video codecs and camera settings. Existing algorithms for recognition of human action in a compressed video have some limitations in this regard, such as (i) requirement of keyframes at a fixed interval, (ii) usage of P frames only, and (iii) normally support single codec only. These limitations are overcome in the proposed method by using arbitrary keyframe intervals, using both P and B frames, and supporting MPEG-4 as well as H.264 codecs. The experimentation is carried out using the benchmark datasets, namely, UCF101, HMDB51, and THUMOS14, and the recognition accuracy in a compressed domain is found to be comparable to that observed in raw video data but at reduced cost of computation. The proposed MVViT method has outperformed other recent methods in terms of a lesser (61.0%) number of parameters and (63.7%) Giga Floating Point Operations Per Second (GFLOPS), while significantly improving accuracy by 0.8%, 5.9% and 16.6% for UCF101, HMDB51 and THUMOS14, respectively. Also, it is observed that the speed is increased by 8% in case of UCF101 when compared to the highest speed reported in the literature on the same dataset. The ablation study of the proposed method has been done using MVViT variants for different codecs and the performance analysis is done in comparison with the state-of-the-art network models.

Commonsense for Zero-Shot Natural Language Video Localization

Zero-shot Natural Language-Video Localization (NLVL) methods have exhibited promising results in training NLVL models exclusively with raw video data by dynamically generating video segments and pseudo-query annotations. However, existing pseudo-queries often lack grounding in the source video, resulting in unstructured and disjointed content. In this paper, we investigate the effectiveness of commonsense reasoning in zero-shot NLVL. Specifically, we present CORONET, a zero-shot NLVL framework that leverages commonsense to bridge the gap between videos and generated pseudo-queries via a commonsense enhancement module. CORONET employs Graph Convolution Networks (GCN) to encode commonsense information extracted from a knowledge graph, conditioned on the video, and cross-attention mechanisms to enhance the encoded video and pseudo-query representations prior to localization. Through empirical evaluations on two benchmark datasets, we demonstrate that CORONET surpasses both zero-shot and weakly supervised baselines, achieving improvements up to 32.13% across various recall thresholds and up to 6.33% in mIoU. These results underscore the significance of leveraging commonsense reasoning for zero-shot NLVL.

P487 Automated Scoring of Patient Endoscopy Videos Using Deep Learning Techniques: A Promising Approach for Clinical Trials in Inflammatory Bowel Disease

Abstract Background Accurate assessment of patient endoscopy videos is crucial for various clinical trials, including those related to inflammatory bowel diseases (IBD). However, manual scoring by trained local and central readers can lead to discrepancies, inconsistencies, and logistical challenges. To address these issues, we devised a novel approach utilizing deep learning techniques to automate the analysis, parsing, and scoring of patient endoscopy videos. Our objectives were to reduce human error/variability, accelerate endoscopic scoring by enhancing read accuracy, and lower clinical trial cost. Methods We developed and trained deep learning models using a large dataset of labeled patient endoscopy images. The models were designed to predict the Mayo endoscopic score (MES), which serves as a standardized measure of disease severity in IBD. Our approach developed convolutional neural networks (CNNs) to extract features from raw video data and processed them into meaningful patterns. The models were optimized using transfer learning and validated on external endoscopy video datasets through rigorous testing protocols. Results Using an iterative process, we developed two sets (total 4) of proof-of-concept (POC) Artificial Intelligence (AI) models using frame level datasets from endoscopy videos. First, quality was assessed by the blur detection model, currently performing at > 90% accuracy, and the bowel preparation model, performing at > 95% accuracy compared to the labels provided by human annotators. Second, MES sub-score prediction currently performs with >75% accuracy for 4-level Mayo sub-score prediction and >90% accuracy in predicting advanced MES scores (2 or higher compared with consensus score derived from 3 annotators). Conclusion The presented study highlights the potential of deep learning techniques in revolutionizing the assessment of patient endoscopy videos in clinical trials for IBD. Automating the endoscopic scoring process enables faster, more accurate, and cost-efficient evaluations, ultimately leading to better detection of patient outcomes. Future research directions involve further refining the models to enhance their performance and expand their application to other gastrointestinal diseases.

DUAL-C: Building a “soft error efficient” on-the-fly compression mechanism for raw video data at edge devices

Nowadays, various edge applications play an increasingly important role in our life, in which video data accounts for a dominant portion of the data traffic. Although standard compression methods, such as H.264 and MPEG4, can considerably reduce the wireless bandwidth for transmission energy savings of edge devices, the large volume of raw video streaming data is still buffered in edge device video buffer, constituting non-negligible energy consumption. Beyond energy issues, edge video data has been increasingly applied in safety–critical scenarios, where reliability has also become a critical concern. Unlike the controlled accuracy sacrifice of the lossy compression mechanism, the high information content per bit may render compressed data more sensitive to random transient faults attacking (i.e., soft errors), thereby inducing severe safety hazards in some video processing tasks (e.g., autonomous vehicle collision).Thus, in this paper, we propose a “soft error efficient” on-the-fly compression mechanism named DUAL-C to achieve energy savings in edge device video buffer while maintaining data reliability. Firstly, observing the duplicate feature of upper bits and approximable characteristic of lower bits within the camera sensors generated streaming raw pixel blocks, we design two compressing techniques: DU-C extracts the common duplicate upper-bits while AL-C merely stores one representative approximate lower-bits for all pixels within a block. By combining DU-C and AL-C with the Sampling-based Approximation paradigm (SA), DUAL-C achieves the overall energy consumption reduction of the video buffer by 71.31%. Secondly, to avoid the reliability “dark side” introduced by compression, we conduct a comprehensive error resilience analysis of compressed data and merely harden the error-critical control bits and common upper bits with ECC. Evaluation results show that DUAL-C compressed data exhibits even better error robustness than its uncompressed counterpart with negligible protection overhead.

Acquisition and usage of robotic surgical data for machine learning analysis

BackgroundThe increasing use of robot-assisted surgery (RAS) has led to the need for new methods of assessing whether new surgeons are qualified to perform RAS, without the resource-demanding process of having expert surgeons do the assessment.Computer-based automation and artificial intelligence (AI) are seen as promising alternatives to expert-based surgical assessment. However, no standard protocols or methods for preparing data and implementing AI are available for clinicians. This may be among the reasons for the impediment to the use of AI in the clinical setting.MethodWe tested our method on porcine models with both the da Vinci Si and the da Vinci Xi. We sought to capture raw video data from the surgical robots and 3D movement data from the surgeons and prepared the data for the use in AI by a structured guide to acquire and prepare video data using the following steps: ‘Capturing image data from the surgical robot’, ‘Extracting event data’, ‘Capturing movement data of the surgeon’, ‘Annotation of image data’.Results15 participant (11 novices and 4 experienced) performed 10 different intraabdominal RAS procedures. Using this method we captured 188 videos (94 from the surgical robot, and 94 corresponding movement videos of the surgeons’ arms and hands). Event data, movement data, and labels were extracted from the raw material and prepared for use in AI.ConclusionWith our described methods, we could collect, prepare, and annotate images, events, and motion data from surgical robotic systems in preparation for its use in AI.

High speed human action recognition using a photonic reservoir computer

The recognition of human actions in videos is one of the most active research fields in computer vision. The canonical approach consists in a more or less complex preprocessing stages of the raw video data, followed by a relatively simple classification algorithm. Here we address recognition of human actions using the reservoir computing algorithm, which allows us to focus on the classifier stage. We introduce a new training method for the reservoir computer, based on “Timesteps Of Interest”, which combines in a simple way short and long time scales. We study the performance of this algorithm using both numerical simulations and a photonic implementation based on a single non-linear node and a delay line on the well known KTH dataset. We solve the task with high accuracy and speed, to the point of allowing for processing multiple video streams in real time. The present work is thus an important step towards developing efficient dedicated hardware for video processing.

Real-Time Video Synopsis via Dynamic and Adaptive Online Tube Resizing

Nowadays, with the increased numbers of video cameras, the amount of recorded video is growing. Efficient video browsing and retrieval are critical issues when considering the amount of raw video data to be condensed. Activity-based video synopsis is a popular approach to solving the video condensation problem. However, conventional synopsis methods always consists of complicated and pairwise energy terms that involve a time-consuming optimization problem. In this paper, we propose a simple online video synopsis framework in which the number of collisions of objects is classified first. Different optimization strategies are applied according to different collision situations to maintain a balance among the computational cost, condensation ratio, and collision cost. Secondly, tube-resizing coefficients that are dynamic in different frames are adaptively assigned to a newly generated tube. Therefore, a suitable mapping result can be obtained in order to represent the proper size of the activity in each frame of the synopsis video. The maximum number of activities can be displayed in one frame with minimal collisions. Finally, in order to remove motion anti-facts and improve the visual quality of the condensed video, a smooth term is introduced to constrain the resizing coefficients. Experimental results on extensive videos validate the efficiency of the proposed method.

PyHAPT: A Python-based Human Activity Pose Tracking data processing framework

<h2>Abstract</h2> We propose a novel Python-based Human Activity Pose Tracking data processing framework (PyHAPT). It provides the functionality to efficiently process annotated human pose tracking raw video data collected in unconstrained environments. Besides, PyHAPT provides the functionalities of interpolation to recover the missing joints data and data visualization that gives insights into the spatial–temporal skeletal information. The processed data could be readily used for developing new human activity recognition deep learning models, which could be deployed on mobile service robots.

Extending the Intelligent Driver Model in SUMO and Verifying the Drive Off Trajectories with Aerial Measurements

Connected and automated driving functions are key components for future vehicles. Due to implementation issues and missing infrastructure, the impact of connected and automated vehicles on the traffic flow can only be evaluated in accurate simulations. Simulation of Urban Mobility (SUMO) provides necessary and appropriate models and tools. SUMO contains many car-following models that replicate automated driving, but cannot realistically imitate human driving behavior. When simulating queued vehicles driving off, existing car-following models are neither able to correctly emulate the acceleration behavior of human drivers nor the resulting vehicle gaps. Thus, we propose a time-discrete 2D Human Driver Model to replicate realistic trajectories. We start by combining previously published extensions of the Intelligent Driver Model (IDM) to one generalized model. Discontinuities due to introduced reaction times, estimation errors and lane changes are conquered with new approaches and equations. Above all, the start-up procedure receives more attention than in existing papers. We also provide a first evaluation of the advanced car-following model using 30 minutes of an aerial measurement. This dataset contains three hours of drone recordings from two signalized intersections in Stuttgart, Germany. The method designed for extracting the vehicle trajectories from the raw video data is outlined. Furthermore, we evaluate the accuracy of the trajectories obtained by the aerial measurement using a specially equipped vehicle.

Dense Semantics-Assisted Networks for Video Action Recognition

Most existing action recognition approaches directly leverage the video-level features to recognize human actions from videos. Although these methods have made remarkable progress, the accuracy is still unsatisfied. When the test video involves complex backgrounds and activities, existing methods usually suffer from a significant drop in accuracy. Human action is inherently a high-level concept. Merely applying a video classification model without a detailed semantic understanding of the video content, e.g., objects, scene context, object motions, object interactions, is inadequate to tackle the challenges for action recognition. Fine-level semantic understanding of videos generates elementary semantic concepts from the raw video data, such as the semantics of objects and background regions. It can be employed to bridge the gap between the raw video data and the high-level concept of human actions. In this work, we leverage dense semantic segmentation masks, which encode rich semantic details, provide extra information for the network training, and improve the performance of action recognition. We propose a novel deep architecture which is named as Dense Semantics-Assisted Convolutional Neural Networks (DSA-CNNs) to effectively utilize dense semantic information of video by a bottom-up attention way in the spatial stream, while by the way of branch fusion in the temporal stream. To verify the effectiveness of our approach, we conduct extensive experiments on publicly available datasets – UCF101, HMDB51, and Kinetics. The experimental results demonstrate that our approach substantially improves existing methods and achieves very competitive performance. It also shows that our approach is superior to other related methods that utilize extra information for action recognition.

Automated Video-Based Analysis Framework for Behavior Monitoring of Individual Animals in Zoos Using Deep Learning-A Study on Polar Bears.

Simple SummaryEvery institution that keeps animals under human care must ensure animal welfare. To analyze the state of an animal, various measurements can be performed, such as blood analysis or fur condition scoring. They also need to be observed as often as possible to gain further insight into their behavior. Such observations are performed manually in most cases, which makes them very labor- and time-intensive and prevent them from being performed on a continual basis. We present a camera-based framework that provides automated observation of animals. The system detects individual animals and analyzes their locations, walking paths, and activity. We test the framework on the two polar bears of the Nuremberg Zoo.The monitoring of animals under human care is a crucial tool for biologists and zookeepers to keep track of the animals’ physical and psychological health. Additionally, it enables the analysis of observed behavioral changes and helps to unravel underlying reasons. Enhancing our understanding of animals ensures and improves ex situ animal welfare as well as in situ conservation. However, traditional observation methods are time- and labor-intensive, as they require experts to observe the animals on-site during long and repeated sessions and manually score their behavior. Therefore, the development of automated observation systems would greatly benefit researchers and practitioners in this domain. We propose an automated framework for basic behavior monitoring of individual animals under human care. Raw video data are processed to continuously determine the position of the individuals within the enclosure. The trajectories describing their travel patterns are presented, along with fundamental analysis, through a graphical user interface (GUI). We evaluate the performance of the framework on captive polar bears (Ursus maritimus). We show that the framework can localize and identify individual polar bears with an F1 score of 86.4%. The localization accuracy of the framework is cm, outperforming current manual observation methods. Furthermore, we provide a bounding-box-labeled dataset of the two polar bears housed in Nuremberg Zoo.

A Cross-Attention BERT-Based Framework for Continuous Sign Language Recognition

Continuous sign language recognition&#x00A0;(CSLR) is a challenging task involving various signal processing techniques to infer the sequences of glosses performed by signers. Existing approaches in CSLR typically use multiple input modalities such as the raw video data and the extracted hand images to improve their recognition accuracy. However, the large modality differences make it difficult to define an integrative framework to effectively exchange and combine the knowledge obtained from different modalities such that they can complement each other for improving the framework&#x0027;s robustness against the gesture variations and background noises in CSLR. To address this issue, we propose a novel cross-attention deep learning framework named the CA-SignBERT. This framework utilizes multiple Bidirectional Encoder Representations from Transformers&#x00A0;(BERT) models to analyze the information from different modalities. Among these BERT models, we introduce a special cross-attention mechanism to ensure an efficient inter-modality knowledge exchange. Besides, an innovative weight control module is proposed to dynamically hybridize their outputs. Experimental results reveal that the CA-SignBERT framework attains state-of-the-art performance in four benchmark CSLR datasets.

BQN: Busy-Quiet Net Enabled by Motion Band-Pass Module for Action Recognition.

A rich video data representation can be realized by means of spatio-temporal frequency analysis. In this research study we show that a video can be disentangled, following the learning of video characteristics according to their spatio-temporal properties, into two complementary information components, dubbed Busy and Quiet. The Busy information characterizes the boundaries of moving regions, moving objects, or regions of change in movement. Meanwhile, the Quiet information encodes global smooth spatio-temporal structures defined by substantial redundancy. We design a trainable Motion Band-Pass Module (MBPM) for separating Busy and Quiet-defined information, in raw video data. We model a Busy-Quiet Net (BQN) by embedding the MBPM into a two-pathway CNN architecture. The efficiency of BQN is determined by avoiding redundancy in the feature spaces defined by the two pathways. While one pathway processes the Busy features, the other processes Quiet features at lower spatio-temporal resolutions reducing both memory and computational costs. Through experiments we show that the proposed MBPM can be used as a plug-in module in various CNN backbone architectures, significantly boosting their performance. The proposed BQN is shown to outperform many recent video models on Something-Something V1, Kinetics400, UCF101 and HMDB51 datasets.

Contactless Video-Based Heart Rate Monitoring of a Resting and an Anesthetized Pig.

Simple SummaryContactless physiological monitoring can be important for animal health and well-being. The current study investigated whether heart rate in pigs can be extracted automatically from videos without disturbing the pig and showed that this was possible with 4.69 beats per minute in mean absolute error. The study also tested different body regions and found that the abdomen was a better region to measure heart rate from videos compared to the front leg or the neck. However, future studies are needed that include videos with different light conditions, different housing systems and multiple pigs to enable real-time on-farm monitoring of heart rate from videos.Heart rate (HR) is a vital bio-signal that is relatively easy to monitor with contact sensors and is related to a living organism’s state of health, stress and well-being. The objective of this study was to develop an algorithm to extract HR (in beats per minute) of an anesthetized and a resting pig from raw video data as a first step towards continuous monitoring of health and welfare of pigs. Data were obtained from two experiments, wherein the pigs were video recorded whilst wearing an electrocardiography (ECG) monitoring system as gold standard (GS). In order to develop the algorithm, this study used a bandpass filter to remove noise. Then, a short-time Fourier transform (STFT) method was tested by evaluating different window sizes and window functions to accurately identify the HR. The resulting algorithm was first tested on videos of an anesthetized pig that maintained a relatively constant HR. The GS HR measurements for the anesthetized pig had a mean value of 71.76 bpm and standard deviation (SD) of 3.57 bpm. The developed algorithm had 2.33 bpm in mean absolute error (MAE), 3.09 bpm in root mean square error (RMSE) and 67% in HR estimation error below 3.5 bpm (PE3.5). The sensitivity of the algorithm was then tested on the video of a non-anaesthetized resting pig, as an animal in this state has more fluctuations in HR than an anaesthetized pig, while motion artefacts are still minimized due to resting. The GS HR measurements for the resting pig had a mean value of 161.43 bpm and SD of 10.11 bpm. The video-extracted HR showed a performance of 4.69 bpm in MAE, 6.43 bpm in RMSE and 57% in PE3.5. The results showed that HR monitoring using only the green channel of the video signal was better than using three color channels, which reduces computing complexity. By comparing different regions of interest (ROI), the region around the abdomen was found physiologically better than the face and front leg parts. In summary, the developed algorithm based on video data has potential to be used for contactless HR measurement and may be applied on resting pigs for real-time monitoring of their health and welfare status, which is of significant interest for veterinarians and farmers.

Knowledge representation and learning of operator clinical workflow from full-length routine fetal ultrasound scan videos.

Ultrasound is a widely used imaging modality, yet it is well-known that scanning can be highly operator-dependent and difficult to perform, which limits its wider use in clinical practice. The literature on understanding what makes clinical sonography hard to learn and how sonography varies in the field is sparse, restricted to small-scale studies on the effectiveness of ultrasound training schemes, the role of ultrasound simulation in training, and the effect of introducing scanning guidelines and standards on diagnostic image quality. The Big Data era, and the recent and rapid emergence of machine learning as a more mainstream large-scale data analysis technique, presents a fresh opportunity to study sonography in the field at scale for the first time. Large-scale analysis of video recordings of full-length routine fetal ultrasound scans offers the potential to characterise differences between the scanning proficiency of experts and trainees that would be tedious and time-consuming to do manually due to the vast amounts of data. Such research would be informative to better understand operator clinical workflow when conducting ultrasound scans to support skills training, optimise scan times, and inform building better user-machine interfaces. This paper is to our knowledge the first to address sonography data science, which we consider in the context of second-trimester fetal sonography screening. Specifically, we present a fully-automatic framework to analyse operator clinical workflow solely from full-length routine second-trimester fetal ultrasound scan videos. An ultrasound video dataset containing more than 200 hours of scan recordings was generated for this study. We developed an original deep learning method to temporally segment the ultrasound video into semantically meaningful segments (the video description). The resulting semantic annotation was then used to depict operator clinical workflow (the knowledge representation). Machine learning was applied to the knowledge representation to characterise operator skills and assess operator variability. For video description, our best-performing deep spatio-temporal network shows favourable results in cross-validation (accuracy: 91.7%), statistical analysis (correlation: 0.98, p < 0.05) and retrospective manual validation (accuracy: 76.4%). For knowledge representation of operator clinical workflow, a three-level abstraction scheme consisting of a Subject-specific Timeline Model (STM), Summary of Timeline Features (STF), and an Operator Graph Model (OGM), was introduced that led to a significant decrease in dimensionality and computational complexity compared to raw video data. The workflow representations were learnt to discriminate between operator skills, where a proposed convolutional neural network-based model showed most promising performance (cross-validation accuracy: 98.5%, accuracy on unseen operators: 76.9%). These were further used to derive operator-specific scanning signatures and operator variability in terms of type, order and time distribution of constituent tasks.

Scene Adaptive Online Surveillance Video Synopsis via Dynamic Tube Rearrangement Using Octree.

Visual surveillance produces a significant amount of raw video data that can be time consuming to browse and analyze. In this work, we present a video synopsis methodology called "scene adaptive online video synopsis via dynamic tube rearrangement using octree (SSOcT)" that can effectively condense input surveillance videos. Our method entailed summarizing the input video by analyzing scene characteristics and determining an effective spatio-temporal 3D structure for video synopsis. For this purpose, we first analyzed the attributes of each extracted tube with respect to scene geometry and complexity. Then, we adaptively grouped the tubes using an online grouping algorithm that exploits these scene characteristics. Finally, the tube groups were dynamically rearranged using the proposed octree-based algorithm that efficiently inserted and refined tubes containing high spatio-temporal movements in real time. Extensive video synopsis experimental results are provided, demonstrating the effectiveness and efficiency of our method in summarizing real-world surveillance videos with diverse scene characteristics.

Automated interpretation of congenital heart disease from multi-view echocardiograms.

Congenital heart disease (CHD) is the most common birth defect and the leading cause of neonate death in China. Clinical diagnosis can be based on the selected 2D key-frames from five views. Limited by the availability of multi-view data, most methods have to rely on the insufficient single view analysis. This study proposes to automatically analyze the multi-view echocardiograms with a practical end-to-end framework. We collect the five-view echocardiograms video records of 1308 subjects (including normal controls, ventricular septal defect (VSD) patients and atrial septal defect (ASD) patients) with both disease labels and standard-view key-frame labels. Depthwise separable convolution-based multi-channel networks are adopted to largely reduce the network parameters. We also approach the imbalanced class problem by augmenting the positive training samples. Our 2D key-frame model can diagnose CHD or negative samples with an accuracy of 95.4%, and in negative, VSD or ASD classification with an accuracy of 92.3%. To further alleviate the work of key-frame selection in real-world implementation, we propose an adaptive soft attention scheme to directly explore the raw video data. Four kinds of neural aggregation methods are systematically investigated to fuse the information of an arbitrary number of frames in a video. Moreover, with a view detection module, the system can work without the view records. Our video-based model can diagnose with an accuracy of 93.9% (binary classification), and 92.1% (3-class classification) in a collected 2D video testing set, which does not need key-frame selection and view annotation in testing. The detailed ablation study and the interpretability analysis are provided. The presented model has high diagnostic rates for VSD and ASD that can be potentially applied to the clinical practice in the future. The short-term automated machine learning process can partially replace and promote the long-term professional training of primary doctors, improving the primary diagnosis rate of CHD in China, and laying the foundation for early diagnosis and timely treatment of children with CHD.