Fast Video Research Articles

Ultrasound video analysis for skill level assessment in FAST ultrasound

ABSTRACT FAST ultrasound is a medical procedure to assess for free fluid following physical trauma. FAST images can often be difficult to interpret and requires operators to be properly trained. Traditionally, skill is assessed by direct observation from experts, which is expensive and error prone. This project aims to use deep learning to provide automated skills assessment for FAST exams. Modified I3D networks, a type of modern neural network with a focus on action-based items, were retrained for this purpose. First, a network to identify the skill level of the users from all the ultrasound videos was trained using FAST videos of each vital region divided by novice, intermediate and expert users. Following this, 4 networks corresponding to skill level identification in each region were trained using the previously constructed model. The model’s performance was evaluated using k-fold cross-validation. Results found a testing accuracy of 82.6% for skills assessment using the modified I3D networks. These results are an improvement over the previous results for skill level evaluation, implying potential use of an I3D network for evaluating skill level from ultrasound video in the future with the proper finetuning.

A Dynamic Frame Selection Framework for Fast Video Recognition.

We introduce AdaFrame, a conditional computation framework that adaptively selects relevant frames on a per-input basis for fast video recognition. AdaFrame, which contains a Long Short-Term Memory augmented with a global memory to provide context information, operates as an agent to interact with video sequences aiming to search over time which frames to use. Trained with policy search methods, at each time step, AdaFrame computes a prediction, decides where to observe next, and estimates a utility, i.e., expected future rewards, of viewing more frames in the future. Exploring predicted utilities at testing time, AdaFrame is able to achieve adaptive lookahead inference so as to minimize the overall computational cost without incurring a degradation in accuracy. We conduct extensive experiments on two large-scale video benchmarks, FCVID and ActivityNet. With a vanilla ResNet-101 model, AdaFrame achieves similar performance of using all frames while only requiring, on average, 8.21 and 8.65 frames on FCVID and ActivityNet, respectively. We also demonstrate AdaFrame is compatible with modern 2D and 3D networks for video recognition. Furthermore, we show, among other things, learned frame usage can reflect the difficulty of making prediction decisions both at instance-level within the same class and at class-level among different categories.

Real-Time Video Stitching for Mine Surveillance Using a Hybrid Image Registration Method

Video stitching technology provides an effective solution for a wide viewing angle monitoring mode for industrial applications. At present, the observation angle of a single camera is limited, and the monitoring network composed of multiple cameras will have many overlapping images captured. Monitoring surveillance cameras can cause the problems of viewing fatigue and low video utilization rate of involved personnel. In addition, current video stitching technology has poor adaptability and real-time performance. We propose an effective hybrid image feature detection method for fast video stitching of mine surveillance video using the effective information of the surveillance video captured from multiple cameras in the actual conditions in the industrial coal mine. The method integrates the Moravec corner point detection and the scale-invariant feature transform (SIFT) feature extractor. After feature extraction, the nearest neighbor method and the random sampling consistency (RANSAC) algorithm are used to register the video frames. The proposed method reduces the image stitching time and solves the problem of feature re-extraction due to the change of observation angle, thus optimizing the entire video stitching process. The experimental results on the real-world underground mine videos show that the optimized stitching method can stitch videos at a speed of 21 fps, effectively meeting the real-time requirement, while the stitching effect has a good stability and applicability in real-world conditions.

AGUnet: Annotation-guided U-net for fast one-shot video object segmentation

The problem of semi-supervised video object segmentation has been popularly tackled by fine-tuning a general-purpose segmentation deep network on the annotated frame using hundreds of iterations of gradient descent. The time-consuming fine-tuning process, however, makes these methods difficult to use in practical applications. We propose a novel architecture called Annotation Guided U-net (AGUnet) for fast one-shot video object segmentation (VOS). AGUnet can quickly adapt a model trained on static images to segmenting the given target in a video by only several iterations of gradient descent. Our AGUnet is inspired by interactive image segmentation, where the interested target is segmented by using user annotated foreground. However, in AGUnet we use a fully-convolutional Siamese network to automatically annotate the foreground and background regions and fuse such annotation information into the skip connection of a U-net for VOS. Our AGUnet can be trained end-to-end effectively on static images instead of video sequences as required by many previous methods. The experiments show that AGUnet runs much faster than current state-of-the-art one-shot VOS algorithms while achieving competitive accuracy, and it has high generalization capability.

Fast Depth Video Coding with Intra Prediction on VVC

Fast Depth Video Coding with Intra Prediction on VVC

Experiments on Flexible Filaments in Air Flow for Aeroelasticity and Fluid-Structure Interaction Models Validation

Several problems in science and engineering are characterized by the interaction between fluid flows and deformable structures. Due to their complex and multidisciplinary nature, these problems cannot normally be solved analytically and experiments are frequently of limited scope, so that numerical simulations represent the main analysis tool. Key to the advancement of numerical methods is the availability of experimental test cases for validation. This paper presents results of an experiment specifically designed for the validation of numerical methods for aeroelasticity and fluid-structure interaction problems. Flexible filaments of rectangular cross-section and various lengths were exposed to air flow of moderate Reynolds number, corresponding to laminar and mildly turbulent flow conditions. Experiments were conducted in a wind tunnel, and the flexible filaments dynamics was recorded via fast video imaging. The structural response of the filaments included static reconfiguration, small-amplitude vibration, large-amplitude limit-cycle periodic oscillation, and large-amplitude non-periodic motion. The present experimental setup was designed to incorporate a rich fluid-structure interaction physics within a relatively simple configuration without mimicking any specific structure, so that the results presented herein can be valuable for models validation in aeroelasticity and also fluid-structure interaction applications.

Teaching at the paramedics school of the City of Munich during the COVID-19 pandemic

Die Corona-Krise hat die Berufsfachschule für Notfallsanitäter der Berufsfeuerwehr München genauso wie alle anderen deutschen Schulen innerhalb weniger Tage gezwungen, den Präsenzunterricht einzustellen. Auch die praktische Ausbildung an den Kliniken sowie auf dem Rettungswagen musste unterbrochen werden. Hohe Ausfallzeiten bei den Auszubildenden drohten und gefährdeten das Ausbildungsziel. Folglich musste Schulunterricht im Homeschooling realisiert werden. Neben der Unterstützung des Münchner Krisenstabs durch Lehrkräfte unserer Abteilung mussten also zusätzlich alle Klassen im virtuellen Klassenzimmer unterrichtet werden, wodurch sich der Arbeitsaufwand der Lehrenden signifikant erhöhte. Im ersten Schritt wurde mit Microsoft Skype eine schnelle Video- und Kommunikationsplattform gefunden. Alles weitere musste aufgrund der Kürze der Zeit im laufenden Unterricht entwickelt werden. Schnell stellte sich heraus, dass Frontalunterricht aufgrund der Monotonie über 35 h pro Woche allein keine Option sein konnte. Darüber hinaus drohten sich fehlende Prüfungsleistungen anzuhäufen, sodass wir auch im Homeschooling Leistungserhebungen sicherstellen mussten. Es zeigte sich innerhalb weniger Tage, dass sich, wenn auch mit gewissen Einschränkungen, nahezu alle Sozialformen und Unterrichtsmethoden im virtuellen Klassenzimmer integrieren lassen. Wie eine Umfrage unter den Auszubildenden zeigte, kann Homeschooling den Unterricht im Klassenzimmer nicht ersetzen, aber im Notfall eine gute Alternative darstellen. Zudem können sich viele Lernende Homeschooling als Ergänzung sehr gut vorstellen. Vielleicht kann die Krise so der Beginn einer neuen ergänzenden Unterrichtsstrategie sein.

Product Quantization Network for Fast Visual Search

Product quantization has been widely used in fast image retrieval due to its effectiveness of coding high-dimensional visual features. By constructing the approximation function, we extend the hard-assignment quantization to soft-assignment quantization. Thanks to the differentiable property of the soft-assignment quantization, the product quantization operation can be integrated as a layer in a convolutional neural network, constructing the proposed product quantization network (PQN). Meanwhile, by extending the triplet loss to the asymmetric triplet loss, we directly optimize the retrieval accuracy of the learned representation based on asymmetric similarity measurement. Utilizing PQN, we can learn a discriminative and compact image representation in an end-to-end manner, which further enables a fast and accurate image retrieval. By revisiting residual quantization, we further extend the proposed PQN to residual product quantization network (RPQN). Benefited from the residual learning triggered by residual quantization, RPQN achieves a higher accuracy than PQN using the same computation cost. Moreover, we extend PQN to temporal product quantization network (TPQN) by exploiting temporal consistency in videos to speed up the video retrieval. It integrates frame-wise feature learning, frame-wise features aggregation and video-level feature quantization in a single neural network. Comprehensive experiments conducted on multiple public benchmark datasets demonstrate the state-of-the-art performance of the proposed PQN, RPQN and TPQN in fast image and video retrieval.

Fast video crowd counting with a Temporal Aware Network

Crowd counting aims to count the number of instantaneous people in a crowded space, and many promising solutions have been proposed for single image crowd counting. With the ubiquitous video capture devices in public safety field, how to effectively apply the crowd counting technique to video content has become an urgent problem. In this paper, we introduce a novel framework based on temporal aware modeling of the relationship between video frames. The proposed network contains a few dilated residual blocks, and each of them consists of the layers that compute the temporal convolutions of features from the adjacent frames to improve the prediction. To alleviate the expensive computation and satisfy the demand of fast video crowd counting, we also introduce a lightweight network to balance the computational cost with representation ability. We conduct experiments on the crowd counting benchmarks and demonstrate its superiority in terms of effectiveness and efficiency over previous video-based approaches.

Numerical and Experimental Studies on the Effect of Surface Roughness and Ultrasonic Frequency on Bubble Dynamics in Acoustic Cavitation

With many emerging applications such as chemical reactions and ultrasound therapy, acoustic cavitation plays a vital role in having improved energy efficiency. For example, acoustic cavitation results in substantial enhancement in the rates of various chemical reactions. In this regard, an applied acoustic field within a medium generates acoustic streaming, where cavitation bubbles appear due to preexisting dissolved gas in the working fluid. Upon cavitation inception, bubbles can undergo subsequent growth and collapse. During the last decade, the studies on the effects of different parameters on acoustic cavitation such as applied ultrasound frequency and power have been conducted. The bubble growth and collapse mechanisms and their distribution within the medium have been classified. Yet, more research is necessary to understand the complex mechanism of multi-bubble behavior under an applied acoustic field. Various parameters affecting acoustic cavitation such as surface roughness of the acoustic generator should be investigated in more detail in this regard. In this study, single bubble lifetime, bubble size and multi-bubble dynamics were investigated by changing the applied ultrasonic field. The effect of surface roughness on bubble dynamics was presented. In the analysis, images from a high-speed camera and fast video recording techniques were used. Numerical simulations were also done to investigate the effect of acoustic field frequency on bubble dynamics. Bubble cluster behavior and required minimum bubble size to be affected by the acoustic field were obtained. Numerical results suggested that bubbles with sizes of 50 µm or more could be aligned according to the radiation potential map, whereas bubbles with sizes smaller than 10 µm were not affected by the acoustic field. Furthermore, it was empirically proven that surface roughness has a significant effect on acoustic cavitation phenomena.

A Shallow Prediction Model with Enhanced Features for Video Time Series

Fast video time series prediction is an important part in video anomaly detection in deep learning framework. Time series deep learning models have good performance in time-series data prediction, such as traffic flow, rainfall and network traffic. The models, such as ConvLSTM, ConvGRU with complex structure and large amount of computation, are difficult to fit the video time series prediction. In this paper, we analyze data characteristics of video sequences and propose a shallow video series prediction model. We not only consider the spatial correlation in continuous video frames, but also investigate the interval video frames, which also maintain spatial correlation. We explore the gray scale pixel values of the interval frames which appear specific trend properties when we connect the context pixels together. Based on the enhanced features, we design a shallow 2D convolutional video time series prediction model with good prediction performance and much less total parameters.

Reliable and Dynamic Appearance Modeling and Label Consistency Enforcing for Fast and Coherent Video Object Segmentation With the Bilateral Grid

We propose a novel optimization framework for video object segmentation, given the initial annotations of objects in the keyframes of an input video sequence. In this work, video data is represented by a Markov Random Field model, and segmentation is achieved by finding the minimum graph cut label assignment. More specifically, we first create a bilateral representation of the input video sequence which reduces the size of the graph that the min-cut must operate on. We then introduce dynamic appearance models to learn the segmentation likelihoods, and the reliability of likelihoods is measured to identify false likelihoods that may cause segmentation errors. Thus, the model accurately describes changes in the object’s appearance that have evolved over time. Furthermore, we augment spatial and temporal connections using a soft higher-order potential, ensuring long-range label consistency in the segmentation. We provide extensive analysis and evaluation with respect to the influence of each component of the framework through the ablation study. Experiments on three benchmark datasets (DAVIS 2016, YouTube-Objects and SegTrack v2) show that our method achieves competitive performance compared to state-of-the-art while having the order of magnitude faster runtime.

Fast and robust video stabilisation with preserved intentional camera motion and smear removal for infrared video

Border military surveillance is one of the demanding and challenging tasks for any nation. Thermal (infrared) camera, which works on the infrared domain, provides complete visual sequences even in pitch dark night conditions. When the video is recorded from a thermal camera mounted on a vehicle, the output video is unstabilised with poor visual quality due to unintentional camera motion. These unwanted motions also introduce smear. Furthermore, there may be situations when a camera is moved intentionally to capture target. This study proposes a fast and robust algorithm for auto stabilisation of videos with smear removal while keeping the intentional motion of camera. This algorithm is developed under the framework of speeded up robust features matching. The proposed algorithm is capable of correcting both motions, i.e. translation as well as rotational. Quality improvement of up to 21 dB is achieved in the stabilised output videos.

Ring decomposition based video copy detection using global ordinal measure features and local features

Visual hashing-based or fingerprinting-based video copy detection approach has been adopted numerously by the video search community due to significant escalation of manipulated copies of original videos over the Internet. Most of the existing video copy detection approaches are robust against the content-preserving distortions such as brightness enhancement and compression, but less robust against the geometric distortions such as rotation and scaling. To mitigate the problem of computation overhead is still challenging in video copy detection. Moreover, there exist a trade-off between discriminability and robustness properties in most of the existing copy detection approaches. In this paper, an effective and fast video copy detection method is presented by exploiting both spatial-temporal information to tackle the above-mentioned challenges. The novelty of proposed method lies in reducing the computation overhead by generating an intermediate candidate database that are similar to the query video using ring-based Ordinal Measure (OM). Then, distinct visual features based on Histogram of Oriented Gradient (HOG) and Singular Value Decomposition (SVD) are extracted from each key-frame of every scenes of the videos of an intermediate candidate database and a query video for copy detection. To avoid the creation of redundant key-frames, the video frames are grouped into different scenes based on Discrete Cosine Transform (DCT). To further preserve the spatial-temporal information, the Temporally Informative Representative Image (TIRI) is used to generate each key-frame of every scenes of videos. The experimental result shows that the proposed method is more efficient and robust against various distortions which outperforms the state-of-the-art copy detection approaches.

Improved motion estimation algorithm based on integrity index and its implementation in x265

With the development of fast motion estimation (ME) algorithms towards video compression standard H.264, burden on integer-pel ME is slightly reduced. However the integer-pel ME is becoming computationally complex due to demand of higher and higher resolution videos. On the other hand, there is always trade-off between the speed and performance of these search algorithms. This work improvises the performance of content awareness enabled integer-pel ME algorithm for encoding the fast and slow motion video sequences of high definition (HD) resolution category viz. 1280×720 and 1920×1080. The algorithm proposes a novel notion of 'integrity index' with focus on increasing the PSNR and reducing the bit rate. It is implemented in the frame work of x265 version 1.7 video encoder. Motion independent ME algorithm is analysed quantitatively in terms of parameters based on rate distortion and ME time. The proposed algorithm is found to be performing better on BD-Rate and BD-PSNR parameters for the videos of both resolutions under consideration. Observed increase in motion estimation time is less than four seconds compared to hexagonal search algorithm of ME which is the benchmark in fast ME algorithm.

Complexity Scalable Bitplane Image Coding With Parallel Coefficient Processing

Very fast image and video codecs are a pursued goal both in the academia and the industry. This paper presents a complexity scalable and parallel bitplane coding engine for wavelet-based image codecs. The proposed method processes the coefficients in parallel, suiting hardware architectures based on vector instructions. Our previous work is extended with a mechanism that provides complexity scalability to the system. Such a feature allows the coder to regulate the throughput achieved at the expense of slightly penalizing compression efficiency. Experimental results suggests that, when using the fastest speed, the method almost doubles the throughput of our previous engine while penalizing compression efficiency by about 10%

Fast Online Video Pose Estimation by Dynamic Bayesian Modeling of Mode Transitions

We propose a fast online video pose estimation method to detect and track human upper-body poses based on a conditional dynamic Bayesian modeling of pose modes without referring to future frames. The estimation of human body poses from videos is an important task with many applications. Our method extends fast image-based pose estimation to live video streams by leveraging the temporal correlation of articulated poses between frames. Video pose estimation is inferred over a time window using a conditional dynamic Bayesian network (CDBN), which we term time-windowed CDBN. Specifically, latent pose modes and their transitions are modeled and co-determined from the combination of three modules: 1) inference based on current observations; 2) the modeling of mode-to-mode transitions as a probabilistic prior; and 3) the modeling of state-to-mode transitions using a multimode softmax regression. Given the predicted pose modes, the body poses in terms of arm joint locations can then be determined more accurately and robustly. Our method is suitable to investigate high frame rate (HFR) scenarios, where pose mode transitions can effectively capture action-related temporal information to boost performance. We evaluate our method on a newly collected HFR-Pose dataset and four major video pose datasets (VideoPose2, TUM Kitchen, FLIC, and Penn_Action). Our method achieves improvements in both accuracy and efficiency over existing online video pose estimation methods.

New Hardware Static and Reconfigurable Architectures for Video Watermarking System

The watermarking technique is an active subject in current research used as a solution for copyright protection in multimedia documents. In this paper, we propose the first hardware invisible robust video watermarking application based on motion estimation. Since the designers of this application face many challenges, two types of architecture are performed: static and dynamic/partial reconfigurable architecture. The proposed architecture is adapted to HEVC encoded video. Two protection techniques are linked up: the digital watermarking to insert a watermark in the video, and the scrambling technique for overall video protection. The watermark embedding is treated in the horizontal and vertical components of even motion vectors. Eventually, the entire vectors are scrambled. The used watermark is a binary sequence where only one bit is inserted into the horizontal and the vertical components of motion vectors. The recommended architecture applies for slow and fast video sequence, where we use a motion estimator reconfigured according to the macro-block video movement. We also utilize a pipeline structure and a clock gating module to increase computing power and reduce power consumption. Experimental results show that the suggested static and dynamic/partial reconfigurable architecture guarantees material efficiency and superior performance in terms of frequency and power consumption.

Recurrent Temporal Aggregation Framework for Deep Video Inpainting.

Video inpainting aims to fill in spatio-temporal holes in videos with plausible content. Despite tremendous progress on deep learning-based inpainting of a single image, it is still challenging to extend these methods to video domain due to the additional time dimension. In this paper, we propose a recurrent temporal aggregation framework for fast deep video inpainting. In particular, we construct an encoder-decoder model, where the encoder takes multiple reference frames which can provide visible pixels revealed from the scene dynamics. These hints are aggregated and fed into the decoder. We apply a recurrent feedback in an auto-regressive manner to enforce temporal consistency in the video results. We propose two architectural designs based on this framework. Our first model is a blind video decaptioning network (BVDNet) that is designed to automatically remove and inpaint text overlays in videos without any mask information. Our BVDNet wins the first place in the ECCV Chalearn 2018 LAP Inpainting Competition Track 2: Video Decaptioning. Second, we propose a network for more general video inpainting (VINet) to deal with more arbitrary and larger holes. Video results demonstrate the advantage of our framework compared to state-of-the-art methods both qualitatively and quantitatively. The codes are available at https://github.com/mcahny/Deep-Video-Inpainting, and https://github.com/shwoo93/video_decaptioning.

Haptic Exploration During Fast Video Playback: Vibrotactile Support for Event Search in Robot Operation Videos

Fast playback allows quick video exploration, but it also decreases the saliency of short events. We propose a haptic exploration for detection of short events during fast video playback, considering that event-related information in vibrotactile feedback can be preserved during fast playback using Time Scale Modification (TSM) methods developed for audio. We evaluate our proposal in two collision detection experiments using first-person view (FPV) videos. In the first experiment, viewers watched at a fixed playback speed, i.e., 1× or 2×, videos recorded with a camera mounted on a platform cart. In this experiment, event-related vibrations were measured at the back of the camera. In the second experiment, viewers used a media controller to adjust the playback speed in videos simulating an exploration with a mobile robot. In this experiment, event-related vibrations were generated from the measurements used in the first experiment. We show that a haptic exploration improves collision awareness under either constant or adjustable playback speed. In both experiments, the number of collisions reported without vibrotactile feedback deviated the greatest from the actual number of collisions in a video. Moreover, collision detection performance with vibrations time-scaled without Time Scale Modification (TSM) methods was not significantly different from performance without vibrations.