Real Video Research Articles

Video characteristics for remote recognition of agonal respiration: A pilot study

AimThe mobile network quality in ambulances can be variable and limited. This pilot study aimed to identify a suitable network setting for recognizing agonal respiration under limited network conditions. MethodsWe recruited five emergency medical technicians, and each participant viewed 30 real-life videos with different resolutions, frame rates, and network scenarios. Thereafter, they reported the respiration pattern of the patient and identified agonal respiration cases. The time at which agonal respiration was identified was also recorded. The answers provided by the five participants were compared with those of two emergency physicians to compare the accuracy and time delay in breathing pattern recognition. ResultsThe overall accuracy for initial respiratory pattern recognition was 80.7% (121/150). The accuracy for normal breathing was 93.3% (28/30), for not breathing was 96% (48/50), and for agonal breathing was 64.3% (45/70). There was no significant difference in successful recognition between video resolutions. However, the rate of time delay in recognizing agonal respiration less than 10 seconds between 15-fps group and 30-fps group had statistical significance (21% vs 52%, p = 0.041). ConclusionThe frame rate emerges as one of critical factors in agonal respiration recognition through telemedicine, outweighing the significance of video resolution.

A Robust Approach to Multimodal Deepfake Detection.

The widespread use of deep learning techniques for creating realistic synthetic media, commonly known as deepfakes, poses a significant threat to individuals, organizations, and society. As the malicious use of these data could lead to unpleasant situations, it is becoming crucial to distinguish between authentic and fake media. Nonetheless, though deepfake generation systems can create convincing images and audio, they may struggle to maintain consistency across different data modalities, such as producing a realistic video sequence where both visual frames and speech are fake and consistent one with the other. Moreover, these systems may not accurately reproduce semantic and timely accurate aspects. All these elements can be exploited to perform a robust detection of fake content. In this paper, we propose a novel approach for detecting deepfake video sequences by leveraging data multimodality. Our method extracts audio-visual features from the input video over time and analyzes them using time-aware neural networks. We exploit both the video and audio modalities to leverage the inconsistencies between and within them, enhancing the final detection performance. The peculiarity of the proposed method is that we never train on multimodal deepfake data, but on disjoint monomodal datasets which contain visual-only or audio-only deepfakes. This frees us from leveraging multimodal datasets during training, which is desirable given their lack in the literature. Moreover, at test time, it allows to evaluate the robustness of our proposed detector on unseen multimodal deepfakes. We test different fusion techniques between data modalities and investigate which one leads to more robust predictions by the developed detectors. Our results indicate that a multimodal approach is more effective than a monomodal one, even if trained on disjoint monomodal datasets.

Enhanced bare-bones particle swarm optimization based evolving deep neural networks

In this research, we propose a variant of the Bare-Bones Particle Swarm Optimization (BBPSO) algorithm for hyper-parameter selection and deep architecture generation for image, audio and video classification tasks. Since the search process of the original BBPSO model is guided by a single leader and the particles’ personal best experiences, there is a lack of interactions pertaining to the neighbouring elite solutions. To overcome this limitation, we propose a versatile search process for a modified BBPSO model that incorporates a number of effective components and operations. These include the neighbouring and global best signals, search actions with Cauchy/Levy scale factors, sub-dimension operations guided by the local and global elite solutions, and a Levy-driven local search mechanism. Moreover, root-finding algorithms are employed which use informative mathematical principles to estimate new root offspring for leader/particle enhancement. A reinforcement learning algorithm is subsequently used to identify the optimal sequential deployment of these numerical analysis methods to increase robustness. Several medical imaging data sets, i.e., ISIC 2017, PH2 and Dermofit skin lesion databases, the ALL-IDB2 microscopic blood image data set, the MURA musculoskeletal radiographic database, the CK + facial expression data set, as well as the Coswara respiratory audio data set and UCF101 video action data set, are employed for evaluation. The proposed BBPSO-optimized Convolutional Neural Network (CNN), bidirectional Long Short-Term Memory (BiLSTM) with attention mechanism, and CNN-BiLSTM models outperform those devised by other PSO and BBPSO variants, as well as state-of-the-art existing studies, significantly, for image, audio respiratory abnormality and realistic video action recognition.

Cell Multi-Bernoulli (Cell-MB) Sensor Control for Multi-Object Search-While-Tracking (SWT).

Information-driven control can be used to develop intelligent sensors that can optimize their measurement value based on environmental feedback. In object tracking applications, sensor actions are chosen based on the expected reduction in uncertainty also known as information gain. Random finite set (RFS) theory provides a formalism for quantifying and estimating information gain in multi-object tracking problems. However, estimating information gain in these applications remains computationally challenging. This paper presents a new tractable approximation of the RFS expected information gain applicable to sensor control for multi-object search and tracking. Unlike existing RFS approaches, the information gain approximation presented in this paper considers the contributions of non-ideal noisy measurements, missed detections, false alarms, and object appearance/disappearance. The effectiveness of the information-driven sensor control is demonstrated through two multi-vehicle search-while-tracking experiments using real video data from remote terrestrial and satellite sensors.

A Novel Framework to Generate Synthetic Video for Foreground Detection in Highway Surveillance Scenarios

Foreground detection (FD) plays an important role in the domain of video surveillance for highway. The design of advanced FD algorithms requires large-scale and diverse video dataset. However, collecting and labeling real dataset is still time-consuming, labor-intensive, and highly subjective. To address this issue, we first use computer graphics (CG) to clone real highway scenarios (HS) and generate synthetic multi-challenge video datasets, called “Synthetic-HS (CG)”, automatically labeled with accurate pixel-level ground truth. The Synthetic-HS (CG) dataset contains eight imaging condition sequences for computer vision research. Then, we design an image translation (IT) model that translates source domain (Synthetic-HS (CG)) to target domain (real). This model uses skip connections and attention module to generate realistic synthetic images “Synthetic-HS (IT)”. We use publicly available Synthetic-HS in combination with the corresponding real video sequence to conduct experiments. The experiment results suggest that: 1) The Synthetic-HS (CG) dataset enables us to provide precise quantitative evaluation of the drawbacks of foreground detection methods 2) The realistic Synthetic-HS (IT) images can be used to promote the visual perception in highway video surveillance.

Zebra Crossing Detection Using Concatenation of SIFT-Gabor Features

Abstract: Zebra-crossing recognition is a challenging activity for visually impaired navigating safely. This paper proposes a zebra-crossing scene recognition model-based fusion of texture features and geometrical features. The Gabor features for 8 orientations are extracted for different orientations Zebra Crossing lines. The SIFT is employed for keypoint detection and description. The final features vector is obtained by concatenating Gabor and SIFT features. The feature vector is optimized using K-means clustering and Principle Component Analysis (PCA). The optimized feature vectors were classified using five Classifiers, SVM, Random Forest, Decision Tree, Naïve Bayes Classifiers. The proposed model successfully recognizes the crosswalks with 89% accuracy. The different classifier models are evaluated for Precision, Accuracy, F1 Score, Recall, and RoC. This paper specifies the classification of two class positive which is zebra-crossing and negative which is no zebra-crossing. The designed system will classify either the crosswalk is detected or not on the basis of percentage specified for each class. This can be done by matching the features of the trained dataset. Those features are matched with the real videos or images to recognize the motion and classify into specific class. More specific crosswalk detection in real time can be executed to get more specific type of positive and negative class and classification.

Design of GPU Network-on-Chip for Real-Time Video Super-Resolution Reconstruction.

Deep learning has a better output quality compared with traditional algorithms for video super-resolution (SR), but the network model needs large resources and has poor real-time performance. This paper focuses on solving the speed problem of SR; it achieves real-time SR by the collaborative design of a deep learning video SR algorithm and GPU parallel acceleration. An algorithm combining deep learning networks with a lookup table (LUT) is proposed for the video SR, which ensures both the SR effect and ease of GPU parallel acceleration. The computational efficiency of the GPU network-on-chip algorithm is improved to ensure real-time performance by three major GPU optimization strategies: storage access optimization, conditional branching function optimization, and threading optimization. Finally, the network-on-chip was implemented on a RTX 3090 GPU, and the validity of the algorithm was demonstrated through ablation experiments. In addition, SR performance is compared with existing classical algorithms based on standard datasets. The new algorithm was found to be more efficient than the SR-LUT algorithm. The average PSNR was 0.61 dB higher than the SR-LUT-V algorithm and 0.24 dB higher than the SR-LUT-S algorithm. At the same time, the speed of real video SR was tested. For a real video with a resolution of 540×540, the proposed GPU network-on-chip achieved a speed of 42 FPS. The new method is 9.1 times faster than the original SR-LUT-S fast method, which was directly imported into the GPU for processing.

INVESTIGATION AND DEVELOPMENT OF METHODS AND ALGORITHMS FOR ANALYZING VIDEO-EEG MONITORING OF DELAYED CEREBRAL ISCHEMIA AFTER SUBARACHNOID HAEMORRHAGE

Abstract. This work aims to study and develop methods and tools for analyzing video-EEG monitoring of delayed cerebral ischemia after subarachnoid haemorrhage. A study was made of methods and algorithms for processing video images, which are supposed to be used to recognize life events and medical care for a patient, which can lead to artefacts in the patient’s EEG records. An approach to video monitoring of the patient was proposed and tested on real video monitoring data using object-oriented logic programming methods in combination with neural network methods for image analysis.

Differentiable Physics Simulation of Dynamics-Augmented Neural Objects

We present a differentiable pipeline for simulating the motion of objects that represent their geometry as a continuous density field parameterized as a deep network. This includes Neural Radiance Fields (NeRFs), and other related models. From the density field, we estimate the dynamical properties of the object, including its mass, center of mass, and inertia matrix. We then introduce a differentiable contact model based on the density field for computing normal and friction forces resulting from collisions. This allows a robot to autonomously build object models that are visually and dynamically accurate from still images and videos of objects in motion. The resulting Dynamics-Augmented Neural Objects (DANOs) are simulated with an existing differentiable simulation engine, Dojo, interacting with other standard simulation objects, such as spheres, planes, and robots specified as URDFs. A robot can use this simulation to optimize grasps and manipulation trajectories of neural objects, or to improve the neural object models through gradient-based real-to-simulation transfer. We demonstrate the pipeline to learn the coefficient of friction of a bar of soap from a real video of the soap sliding on a table. We also learn the coefficient of friction and mass of a Stanford bunny through interactions with a Panda robot arm from synthetic data, and we optimize trajectories in simulation for the Panda arm to push the bunny to a goal location.

Surface deformation tracking in monocular laparoscopic video.

Image-guided surgery has been proven to enhance the accuracy and safety of minimally invasive surgery (MIS). Nonrigid deformation tracking of soft tissue is one of the main challenges in image-guided MIS owing to the existence of tissue deformation, homogeneous texture, smoke and instrument occlusion, etc. In this paper, we proposed a piecewise affine deformation model-based nonrigid deformation tracking method. A Markov random field based mask generation method is developed to eliminate tracking anomalies. The deformation information vanishes when the regular constraint is invalid, which further deteriorates the tracking accuracy. Atime-series deformation solidification mechanism is introduced to reduce the degradation of the deformation field of the model. For the quantitative evaluation of the proposed method, we synthesized nine laparoscopic videos mimicking instrument occlusion and tissue deformation. Quantitative tracking robustness was evaluated on the synthetic videos. Three real videos of MIS containing challenges of large-scale deformation, large-range smoke, instrument occlusion, and permanent changes in soft tissue texture were also used to evaluate the performance of the proposed method. Experimental results indicate the proposed method outperforms state-of-the-art methods in terms of accuracy and robustness, which shows good performance in image-guided MIS.

EGRA-NeRF: Edge-Guided Ray Allocation for Neural Radiance Fields

Recently, Neural Radiance Fields (NeRF) has demonstrated great potential in synthesizing novel views for realistic video generation. However, renderings from NeRF appear excessively blurred and contain aliasing artifacts in some textures or edges. To alleviate this problem, Edge-Guided Ray Allocation (EGRA-NeRF) module is proposed in this paper. Such method proposes a novel ray allocation strategy to focus more rays on the textures and edges of the scene during the training stage. Specially, the Canny edge detector is introduced to generate the ray guidance map. This guidance map can lead the model to find areas with more textures. To focus more rays on edges and textures, a method is explored to re-adjust the ray guidance map and re-allocate rays dynamically. Compared with the ray allocation in most existing NeRF-based methods, EGRA-NeRF focuses rays more on textures and edges, which can achieve the refinement of edges and texture details. Experiments show that NeRF-based algorithms (NeRF and Instant-NGP) with our EGRA module can achieve more realistic video generation, while maintaining almost the same computation time.

PEDESTRIAN DETECTION AND TRACKING OF THEIR MOVEMENT TRAJECTORY USING THE BACKGROUND SEGMENTATION METHOD BASED ON KNN

Problem statement. Target detection and video image tracking is one of the important topics of computer vision, as well as a problem that needs to be urgently addressed in practical applications. Interference makes it difficult to get the target position. Faced with this problem, scientists have proposed many tracking algorithms. Purpose. In real video monitoring, the system can automatically detect the foreground and draw the trajectory of the foreground. Methods. Use the KNN background segmenting algorithm in combination with OpenCV to detect the foreground and track the trajectory of the video. Novelty. It can continuously detect the foreground in the video and is also applicable to the new foreground in the video. This method is easy to call, does not require the use of a large amount of computer performance resources and can achieve real-time detection and tracking. Result. In a real test, we got good test results, we successfully identified moving pedestrians on video and drew their trajectories. Practical relevance. The algorithm can be applied to road traffic, can determine the trajectory of a vehicle to track vehicles, and can also be used to detect pedestrians to pave the way for subsequent recognition of pedestrian behavior.

Communication, Computing, and Caching Trade-Off in VR Networks

As technology continues to advance, virtual reality (VR) video services are able to provide an increasingly realistic video experience. VR applications are limited, since the creation of an immersive experience requires processing and delivery of incredibly huge amounts of data. A potential technique to decrease the operation time for VR, as well as its energy use, is mobile edge computing (MEC). In this study, we develop a VR network in which several MEC servers can supply field-of-view (FOV) files to a VR device in order to satisfy the transmission requirements of VR video service and improve the quality of the experience. In this way, the projection process from 2D FOV to 3D FOV and the cached data is possible on an MEC server or a VR device. A cooperative computational offloading and caching strategy is developed as a decision matrix to reduce transmission requirements based on the service time constraint requirement. The VR video service mechanism is examined through the decision matrix. The trade-off between communication, caching, and computation (3C trade-off) is further implemented by means of a closed equation for the decision matrix. Results from simulations show that the suggested technique can perform close to optimally compared to alternative opposing methods.

A Study of Using Synthetic Data for Effective Association Knowledge Learning

Association, aiming to link bounding boxes of the same identity in a video sequence, is a central component in multi-object tracking (MOT). To train association modules, e.g., parametric networks, real video data are usually used. However, annotating person tracks in consecutive video frames is expensive, and such real data, due to its inflexibility, offer us limited opportunities to evaluate the system performance w.r.t. changing tracking scenarios. In this paper, we study whether 3D synthetic data can replace real-world videos for association training. Specifically, we introduce a large-scale synthetic data engine named MOTX, where the motion characteristics of cameras and objects are manually configured to be similar to those of real-world datasets. We show that, compared with real data, association knowledge obtained from synthetic data can achieve very similar performance on real-world test sets without domain adaption techniques. Our intriguing observation is credited to two factors. First and foremost, 3D engines can well simulate motion factors such as camera movement, camera view, and object movement so that the simulated videos can provide association modules with effective motion features. Second, the experimental results show that the appearance domain gap hardly harms the learning of association knowledge. In addition, the strong customization ability of MOTX allows us to quantitatively assess the impact of motion factors on MOT, which brings new insights to the community.

Self scale estimation of the tracking window merged with adaptive particle filter tracker

Tracking a mobile object is one of the important topics in pattern recognition, but style has some obstacles. A Reliable tracking system must adjust their tracking windows in real time according to appearance changes of the tracked object. Furthermore, it has to deal with many challenges when one or multiple objects need to be tracked, for instance when the target is partially or fully occluded, background clutter, or even some target region is blurred. In this paper, we will present a novel approach for a single object tracking that combines particle filter algorithm and kernel distribution that update its tracking window according to object scale changes, whose name is multi-scale adaptive particle filter tracker. We will demonstrate that the use of particle filter combined with kernel distribution inside the resampling process will provide more accurate object localization within a research area. Furthermore, its average error for target localization was significantly lower than 21.37 pixels as the mean value. We have conducted several experiments on real video sequences and compared acquired results to other existing state of the art trackers to demonstrate the effectiveness of the multi-scale adaptive particle filter tracker.

Smart Traffic Light Using YOLO Based Camera with Deep Reinforcement Learning Algorithm

Congestion is a common problem that often occurs in big cities. Congestion causes a lot of losses, such as in terms of time, economy, to the psychology of road users. One of the causes of congestion is traffic lights that are not adaptive to the dynamics of traffic flow. This final project tries to solve this problem using a Reinforcement Learning approach combined with a SUMO (Simulation of Urban Mobility) traffic simulator. The data used is the real video data of the KD Cowek intersection, Surabaya. The video data is processed using the YOLO algorithm which will detect and count vehicles. The output of the video processing will be used in Reinforcement Learning. The result of Reinforcement Learning is that the total length of the traffic queue at 06.00 – 09.00 has an average of 106 vehicles.

Shapes of You? Investigating the Acceptance of Video-Based AAL Technologies Applying Different Visualization Modes.

An aged population, increasing care needs, and a lack of (in)formal caregivers represent major challenges for our society today. Addressing these challenges fuels efforts and developments in innovative technologies leading to various existing AAL applications aiming at improving autonomy, independence, and security in older age. Here, the usage of video-based AAL technologies is promising as detailed information can be obtained and analyzed. Simultaneously, this type of technology is strongly connected with privacy concerns due to fears of unauthorized data access or inappropriate use of recorded data potentially resulting in rejection and non-use of the applications. As privacy-preserving visualizations of video data can diminish those concerns, this empirical study examines the acceptance and privacy perceptions of video-based AAL technology applying different visualization modes for privacy preservation (n = 161). These visualization modes differed in their degrees of visibility and identifiability, covering different levels of privacy preservation (low level: "Blurred" mode; medium level: "Pixel" and "Grey" modes; high level: "Avatar" mode) and are specifically evaluated based on realistic video sequences. The results of our study indicate a rather low acceptance of video-based AAL technology in general. From the diverse visualization modes, the "Avatar" mode is most preferred as it is perceived as best suitable to protect and preserve the users' privacy. Beyond that, distinct clusters of future users were identified differing in their technology evaluation as well as in individual characteristics (i.e., privacy perception, technology commitment). The findings support the understanding of potential users' needs for a successful future design, development, and implementation of video-based, but still privacy-preserving AAL technology.

A mutually boosting dual sensor computational camera for high quality dark videography

Videos captured under low light conditions suffer from severe noise. A variety of efforts have been devoted to image/video noise suppression and made large progress. However, in extremely dark scenarios, extensive photon starvation would hamper extracting latent structures buried in noise. Instead, developing an imaging system collecting more photons is a more effective way for high-quality video capture in dark environments. In this paper, we propose to build a dual-sensor camera to additionally collect the photons in NIR wavelength, and make use of the correlation between RGB and near-infrared (NIR) spectrum to perform high-quality reconstruction from noisy dark video pairs. In hardware, we build a compact dual-sensor camera capturing parallax-free RGB and NIR videos simultaneously. Computationally, we propose a dual-channel multi-frame attention network (DCMAN) utilizing spatial–temporal-spectral priors to mutually boost the quality of low-light RGB and NIR videos. In addition, we build a high-quality paired RGB and NIR video dataset, based on which the approach can be applied to different sensors easily by training the DCMAN model with simulated noisy input following a physical-process-based CMOS noise model. Both experiments on synthetic and real videos validate the performance of this compact dual-sensor camera design and the corresponding reconstruction algorithm in dark videography.

Unsupervised Learning-Based Framework for Deepfake Video Detection

With the continuous development of computer hardware equipment and deep learning technology, it is easier for people to swap faces in videos by currently-emerging multimedia tampering tools, such as the most popular deepfake. It would bring a series of new threats of security. Although many forensic researches have focused on this new type of manipulation and achieved high detection accuracy, most of which are based on supervised learning mechanism with requiring a large number of labeled samples for training. In this paper, we first develop a novel unsupervised detection manner for identifying deepfake videos. The main fundamental behind our proposed method is that the face region in the real video is taken by the camera while its counterpart in the deepfake video is usually generated by the computer; the provenance of two videos is totally different. Specifically, our method includes two clustering stages based on Photo-Response Non-Uniformity (PRNU) and noiseprint feature. Firstly, the PRNU fingerprint of each video frame is extracted, which is used to cluster the full-size identical source video (regardless of its real or fake). Secondly, we extract the noiseprint from the face region of the video, which is used to identify (re-cluster for the task of binary classification) the deepfake sample in each cluster. Numerical experiments verify our proposed unsupervised method performs very well on our own dataset and the benchmark FF++ dataset. More importantly, its performance rivals that of the supervised-based state-of-the-art detectors.

Toward Deep Transfer Learning for Realistic Activity Recognition in Videos

Today, videos have become popular on the internet and specified in social network and media platforms such as Youbue, Ticktok, and Vimeo. Video understanding has attracted much attention in the research community in recent years. Automatically recognizing human activity in wild videos is a trending research topic with a wide range of applications in advertising, smarthome, and surveillance camera systems. Deep convolutional neural networks have become a new de facto visual recognition problem. It achieved much success in image recognition problems that are leveraged from the ImageNet dataset. Many researchers have applied CNNs to the video domain, but the results in realistic video still have many challenges, and the recognition rate is not as expected. Because the realistic activity in video is extremely small, we cannot train a large deep convolutional network to achieve good performance. In this work, we answer the question “how could we transfer the visual features from the ImageNet dataset into video for activity recognition tasks?”. We propose an approach based on the pretrained models from ImageNet for activity recognition in video that is based on transforming video into an image map for temporal information and a frame-based description for spatial information. We test our proposed approach on three datasets, UCF11, UCF50 and UCF101, and it achieves an accuracy of over 95% for the backbone of ResNet and over 88% for the backbone of MobileNet. The experimental results show that our proposed method is robust and efficient in wild video datasets.