Real Video Frames Research Articles

Attention-guided generator with dual discriminator GAN for real-time video anomaly detection

Detecting anomalies in videos presents a significant challenge in the field of video surveillance. The primary goal is identifying and detecting uncommon actions or events within a video sequence. The difficulty arises from the limited availability of video frames depicting anomalies and the ambiguous definition of anomaly. Based on extensive applications of Generative Adversarial Networks (GANs), which consist of a generator and a discriminator network, we propose an Attention-guided Generator with Dual Discriminator GAN (A2D-GAN) for real-time video anomaly detection (VAD). The generator network uses an encoder–decoder architecture with a multi-stage self-attention added to the encoder and multi-stage channel attention added to the decoder. The framework uses adversarial learning from noise and video frame reconstruction to enhance the generalization of the generator network. Also, of the dual discriminator in A2D-GAN, one discriminates between the reconstructed video frame and the real video frame, while the other discriminates between the reconstructed noise and the real noise. Exhaustive experiments and ablation studies on four benchmark video anomaly datasets, namely UCSD Peds, CUHK Avenue, ShanghaiTech, and Subway, demonstrate the effectiveness of the proposed A2D-GAN compared to other state-of-the-art methods. The proposed A2D-GAN model is robust and can detect anomalies in videos in real-time. The source code to replicate the results of the proposed A2D-GAN model is available at https://github.com/Rituraj-ksi/A2D-GAN.

P–029 Identification of spermatozoa by unsupervised learning from video data

Abstract Study question Can artificial intelligence (AI) algorithms identify spermatozoa in a semen sample without using training data annotated by professionals? Summary answer Unsupervised AI methods can discriminate the spermatozoon from other cells and debris. These unsupervised methods may have a potential for several applications in reproductive medicine. What is known already Identification of individual sperm is essential to assess a given sperm sample’s motility behaviour. Existing computer-aided systems need training data based on annotations by professionals, which is resource demanding. On the other hand, data analysed by unsupervised machine learning algorithms can improve supervised algorithms that are more stable for clinical applications. Therefore, unsupervised sperm identification can improve computer-aided sperm analysis systems predicting different aspects of sperm samples. Other possible applications are assessing kinematics and counting of spermatozoa. Study design, size, duration Three sperm-like paint images were manipulated using a graphic design tool and used to train our AI system. Two paintings have an ash colour background and randomly distributed white colour circles, and one painting has a predefined pattern of circles. Selected semen sample videos from a public dataset with videos obtained from 85 participants were used to test our AI system. Participants/materials, setting, methods Generative adversarial networks (GANs) have become common AI methods to process data in an unsupervised way. Based on single image frames extracted from videos, a GAN (SinGAN) can be trained to determine and track locations of sperms by translating the real images into localization paintings. The resulting model showed the potential of identifying the presence of sperms without any prior knowledge about data. Main results and the role of chance Visual comparisons of localization paintings to real sperm images show that inverse training of SinGANs can track sperms. Converting colour frames into grayscale frames and using grayscale synthetic sperm-like frames showed the best visual quality of generated localization paintings of sperm frames. Feeding real sperm video frames to the SinGAN at different scaling factors, which is defining the resolution of the input image, showed different quality levels of generated sperm localization paintings. A sperm frame given to the algorithm with a scaling factor of one leads to random sperm tracking, while the scales two to four result in more accurate localization maps than scaling levels five to eight. In contrast, scales from six to eight result in an output close to the input frame. The proposed method is robust in terms of the number of spermatozoa, meaning that the detection works well for samples with a low or high sperm count. For visual comparisons, visit our Github page: https://vlbthambawita.github.io/singan-sperm/. The sperm tracking speed of our SinGAN using an NVIDIA 1080 graphic processing unit, is around 17 frames per second, which can be improved by using parallel video processing capabilities. This shows the capability of using this method for real-time analysis. Limitations, reasons for caution Unsupervised methods are hard to train, and the results need human verification. The proposed method will need quality control and must be standardized. Unsupervised sperm tracking SinGAN may identify blurry bright spots as non-existing sperm heads which may restrict the use of SinGAN sperm tracking for sperm counting. Wider implications of the findings: Assessment of semen samples according to the WHO guidelines is subjective and resource-demanding. This unsupervised model might be used to develop new systems for less time-consuming and more accurate evaluation of semen samples. It may also be used for real-time analysis of prepared spermatozoa for use in assisted reproduction technology. Trial registration number N/A

A Video Prediction Method Based on Optical Flow Estimation and Pixel Generation

Video prediction has developed rapidly after the booming of deep learning. As an important part of unsupervised representation learning, it plays an important role in anomalous behavior detection, autonomous driving, video games, and other fields. However, the prediction method based on optical flow estimation is susceptible to brightness change and camera shake, and it is difficult to predict occluded objects. While the prediction method based on pixel generation is difficult to fit ambiguous and complex scenes, which leads to a blurry prediction. In this work, we proposed an end-to-end video prediction framework that combines the optical flow estimation module with the pixel generation module by a learnable mask weight to predict high-fidelity videos. To further improve the prediction effect, we introduce adversarial training to the framework. We introduced a frame discriminator and a sequence discriminator to ensure the consistency of the spatio-temporal distribution of predicted video frames and real video frames. The results of experiments on challenging datasets demonstrate the practicability and effectiveness of our proposed video prediction framework. On the one hand, our proposed framework has achieved an equal quality compared with the current latest model, which requires fewer parameters and has a faster prediction speed. On the other hand, the results of ablation experiments demonstrate the effect of fusing different modules and the effectiveness of adversarial training.

A DCP-based Method for Improving Laparoscopic Images.

Laparoscopy is an invasive surgical technique performed in abdominal surgery that provides faster recovery than conventional open surgeries. It requires to introduce a camera to observe the surgical maneuvers. However, during this intervention, the quality of the image may be reduced due to the creation of water vapor and carbon dioxide inside the pelvic-abdominal cavity. This phenomenon produces a nebulous image that causes interruptions during the surgical intervention. Removing this nebulous effect is a key factor to improve the vision of the surgeon. In this study, we have used a method based on the dark channel prior to remove the haze in video frames of laparoscopic surgeries to provide better quality images. The results have been positively evaluated by specialists using real video frames of laparoscopic surgeries, thus demonstrating that this method can be effective in improving the quality of the images without losing any detail of the original image.

SAR Analysis for Real Video Super- Resolution Improvement

This paper presents a simultaneous autoregressive (SAR) analysis method to describe the unknown signal-to-noise ratio (SNR) and the texture feature of low-quality real video frames when the ground-truth images are not available. The real video images degraded by the factors, such as electronic noise, oversaturated pixels, motion blur, and compression artifacts, often result in poor motion registration estimation, which makes the performance of the existing video super-resolution (VSR) algorithms lower than expected. It is hard to estimate the SNR of the low-quality real frames without any prior knowledge. To solve this problem, we made a connection between SAR hyperparameters and the SNR of real images. The relationship expression of them was given in this paper. Using the proposed method, well-registered low-quality real video frames can be selected to decrease the root mean squared error (RMSE) of motion estimation of video frames for VSR reconstruction improvement. The anomalous low-quality frame images whose SAR hyperparameters values are inconsistent with others will be considered for removal. Synthetic experiments were designed to illustrate how the SAR hyperparameters values vary with the variation of synthetic parameters. In order to better illustrate the effectiveness of the proposed method, real low-quality videos captured under different conditions were tested under VSR reconstruction experiments. The VSR reconstruction results show that the results obtained using SAR prior analysis have sharper edges and fewer ringing artifacts than the original results. It indicates that the proposed method is helpful to obtain better results of motion registration estimation for low-quality real video images.

Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy

PurposeColorectal cancer is the third most common cancer worldwide, and early therapeutic treatment of precancerous tissue during colonoscopy is crucial for better prognosis and can be curative. Navigation within the colon and comprehensive inspection of the endoluminal tissue are key to successful colonoscopy but can vary with the skill and experience of the endoscopist. Computer-assisted interventions in colonoscopy can provide better support tools for mapping the colon to ensure complete examination and for automatically detecting abnormal tissue regions.MethodsWe train the conditional generative adversarial network pix2pix, to transform monocular endoscopic images to depth, which can be a building block in a navigational pipeline or be used to measure the size of polyps during colonoscopy. To overcome the lack of labelled training data in endoscopy, we propose to use simulation environments and to additionally train the generator and discriminator of the model on unlabelled real video frames in order to adapt to real colonoscopy environments.ResultsWe report promising results on synthetic, phantom and real datasets and show that generative models outperform discriminative models when predicting depth from colonoscopy images, in terms of both accuracy and robustness towards changes in domains.ConclusionsTraining the discriminator and generator of the model on real images, we show that our model performs implicit domain adaptation, which is a key step towards bridging the gap between synthetic and real data. Importantly, we demonstrate the feasibility of training a single model to predict depth from both synthetic and real images without the need for explicit, unsupervised transformer networks mapping between the domains of synthetic and real data.

Multi-frame image super-resolution reconstruction using sparse co-occurrence prior and sub-pixel registration

In this paper, a hybrid super-resolution (SR) method is proposed by combining the concepts of both multi-frame and single-frame SR to generate a high-resolution (HR) image. The main contributions are in two aspects: the first one is hierarchical iterative sub-pixel registration, which provides accurate registration information of input low-resolution (LR) images or frames, to generate an initial HR image; the second one is to enhance the initial HR image with sparse co-occurrence prior, resulted from specially-designed dictionaries containing patches from both generic training images and interpolated input LR images. As a whole, the proposed hybrid SR method makes use of information from both sub-pixel registration and sparse co-occurrence prior to get reconstructed SR image with large zoom-in factor. The simulation results from synthetic images and real video frames illustrate its effectiveness and the superiority in image quality over conventional multi-frame and single-frame SR methods.

Ordinal Regression Based Subpixel Shift Estimation for Video Super-Resolution

We present a supervised learning-based approach for subpixel motion estimation which is then used to perform video super-resolution. The novelty of this work is the formulation of the problem of subpixel motion estimation in a ranking framework. The ranking formulation is a variant of classification and regression formulation, in which the ordering present in class labels namely, the shift between patches is explicitly taken into account. Finally, we demonstrate the applicability of our approach on superresolving synthetically generated images with global subpixel shifts and enhancing real video frames by accounting for both local integer and subpixel shifts.

A modular approach to facial feature segmentation on real sequences

In this paper a modular approach of gradual confidence for facial feature extraction over real video frames is presented. The problem is being dealt under general imaging conditions and soft presumptions. The proposed methodology copes with large variations in the appearance of diverse subjects, as well as of the same subject in various instances within real video sequences. Areas of the face that statistically seem to be outstanding form an initial set of regions that are likely to include information about the features of interest. Enhancement of these regions produces closed objects, which reveal—through the use of a fuzzy system—a dominant angle, i.e. the facial rotation angle . The object set is restricted using the dominant angle. An exhaustive search is performed among all candidate objects, matching a pattern that models the relative position of the eyes and the mouth. Labeling of the winner features can be used to evaluate the features extracted and provide feedback in an iterative framework. A subset of the MPEG-4 facial definition or facial animation parameter set can be obtained. This gradual feature revelation is performed under optimization for each step, producing a posteriori knowledge about the face and leading to a step-by-step visualization of the features in search.

Optimizing block size in motion-compensated video coding

Despite the widespread experience with block-based video coders, there is little analysis or theory that quantitatively explains the effect of block size on encoding bit rate, and ordinarily the block size for a coder is chosen based on empirical experiments on video sequences of interest. In this work, we derive a procedure to determine the optimal block size that minimizes the encoding rate for a typical block-based video coder. To do this, we analytically model the effect of block size and derive expressions for the encoding rates for both motion vectors and difference frames as functions of block size. Minimizing these expressions leads to a simple formula that indicates how to choose the block size in these types of coders. This formula also shows that the best block size is a function of the accuracy with which the motion vectors are encoded and several parameters related to key characteristics of the video scene, such as image texture, motion activity, interframe noise, and coding distortion. We implement the video coder and use our analysis to optimize and explain its performance on real video frames.

Motion segmentation by multistage affine classification

We present a multistage affine motion segmentation method that combines the benefits of the dominant motion and block-based affine modeling approaches. In particular, we propose two key modifications to a recent motion segmentation algorithm developed by Wang and Adelson (1994). 1) The adaptive k-means clustering step is replaced by a merging step, whereby the affine parameters of a block which has the smallest representation error, rather than the respective cluster center, is used to represent each layer; and 2) we implement it in multiple stages, where pixels belonging to a single motion model are labeled at each stage. Performance improvement due to the proposed modifications is demonstrated on real video frames.

Methods for Optimal Guidance of Industrial Robot Motions

Industrial robots are used in a number of very different tasks. Typical applications are spot welding, arc welding, assembly, painting, material handling. Therefore application flexibility is a key concept for an industrial robot. Application flexibility of a robot is greatly enhanced by the use of fast and easy methods of teaching the robot. This paper deals with a new man-machine interface for teaching robots. At our institute a colour-screen-system with light pen has been developed. This system is used for the definition of the work flow at workpiece handling from a moving conveyor to the place of deposit. With this system the operator can draw instructions over a real video frame showing the robot and the working place. Possible instructions are obstacles or intermediate positions of the robot path. A first realization was done with one camera at the ceiling.A further crucial point treated in this paper is the calculation of robot paths. Depending on intermediate path points, forbidden areas and given velocities the paths of robot motions are computed. The computation algorithm gets its input data from the teaching informations as well as from an optical sensor, which recognizes position and orientation of a workpiece on a moving conveyor. Because the calculation of robot paths is done in joint coordinates transformations from “world ”-coordinate systems (teaching and sensor coordinate systems) into the joint coordinate system are necessary. For the trajectories of the robot motions polynomials are determined. Determining these polynomials various optimisation criteria (length, time, energy consumption, minimum stress imposed on material) can be taken into consideration. The calculations for the optimal guidance of robots are done by a microprocessor, which is a part of a distributed microprocessor system.