Frame Motion Research Articles

Non-Uniform Motion Aggregation with Graph Convolutional Networks for Skeleton-Based Human Action Recognition

Skeleton-based human action recognition aims to recognize human actions from given skeleton sequences. The literature utilizes fixed-stride sampling and uniform aggregations, which are independent of the input data and do not focus on representative motion frames. In this paper, to overcome the challenge of the fixed uniform aggregation strategy being unable to focus on discriminative motion information, a novel non-uniform motion aggregation embedded with a graph convolutional network (NMA-GCN) is proposed for skeleton-based human action recognition. Based on the skeleton quality and motion-salient regions, NMA is able to focus on the discriminative motion information of human motion-salient regions. Finally, the aggregated skeleton sequences are embedded with the GCN backbone for skeleton-based human action recognition. Experiments were conducted on three large benchmarks: NTU RGB+D, NTU RGB+D 120, and FineGym. The results show that our method achieves 93.4% (Xsub) and 98.2% (Xview) on NTU RGB+D dataset, 87.0% (Xsub) and 90.0% (Xset) on the NTU RGB+D 120 dataset, and 90.3% on FineGym dataset. Ablation studies and evaluations across various GCN-based backbones further support the effectiveness and generalization of NMA-GCN.

Research on Three-Dimensional Shape Curve Reconstruction Technology for a Scraper Conveyor on an Intelligent Working Face.

"Three straight and two flat" is the inevitable demand when realizing the intelligent mining of a fully mechanized mining face. To address the crucial technical issue of lacking accurate perception of the shape of the scraper conveyor during intelligent coal mining, a three-dimensional curvature sensor involving fiber Bragg grating (FBG) is used as a perceptive tool to conduct curve reconstruction research based on different local motion frames and to reconstruct the shape of the scraper conveyor. Firstly, the formation process of the 'S'-shaped bending section of the scraper conveyor during the pushing process is determined. Based on the FBG sensing principle, a mathematical model between the variation in the central wavelength and the strain and curvature is established, and the cubic B-spline interpolation method is employed to continuously process the obtained discrete curvature. Secondly, based on differential geometry, a spatial curve reconstruction algorithm based on the Frenet moving frame is derived, and the shape curve prediction interpolation model is built based on a gated recurrent unit (GRU) model, which reduces the impact of the decrease in curve reconstruction accuracy caused by damage to some grating measuring points. Finally, an experimental platform was designed and built, and sensors with curvature radii of 6 m, 7 m, and 8 m were tested. The experimental results showed that the reconstructed curve was essentially consistent with the actual shape, and the absolute error at the end was about 2 mm. The feasibility of this reconstruction algorithm in engineering has been proven, and this is of great significance in achieving shape curve perception and straightness control for scraper conveyors.

Ai-Based Building Security System Using Vision Tracking Motion Method

Security is something that is needed by every human being, because with security there will be an impression of comfort. Especially in a place of residence or a place to carry out economic transactions or even an agency. Artificial intelligence is a form of scientific discipline which is an artificial intelligence that functions to facilitate the process of human life technologically. Machine Learning is a machine that was developed to be able to learn by itself without direction from the user. Video tracking motion is a method that is used as a reference to capture movement in objects that change position which is translated on the process frame motion system. An application related to Artificial Intelligence (AI) which can control automatically regarding the video camera system in the form of IPCam and Analog UsbCam, with a database as a medium for storing the results of video capture images regarding process frames that exceed the limit, which functions for security in a place in the form of house or building.

Automatic highlight detection in videos of martial arts tricking

We propose a novel strategy for the automatic detection of highlight events in user-generated tricking videos, to the best of our knowledge, the first one specifically tailored for this complex sport. Most current methods for related sports leverage high-level semantics such as predefined camera angles or common editing practices, or rely on depth cameras to achieve automatic detection. However, our approach only relies on the contents (themselves) in the frames of a given video, and consists in a four stage pipeline. The first stage identifies foreground key points of interest along with an estimation of their motion in the video frames. In the second stage, these points are grouped into regions of interest based on their proximity and motion. Their behavior over time is evaluated in the third stage to generate an attention map indicating the regions participating in the most relevant events. The fourth and final stage provides the extracted video sequences where highlights have been identified. Experimental results attest to the effectiveness of our approach, which shows high recall and precision values at frame level, with detections that fit well the ground truth events.

Lagrangian reduction and wave mean flow interaction

How does one derive models of dynamical feedback effects in multiscale, multiphysics systems such as wave mean flow interaction (WMFI)? We shall address this question for hybrid dynamical systems, defined as systems whose motion can be expressed as the composition of two or more Lie-group actions. Hybrid systems abound in fluid dynamics. Examples include: the dynamics of complex fluids such as liquid crystals; wind-driven waves propagating with the currents moving on the sea surface; turbulence modelling in fluids and plasmas; and classical–quantum hydrodynamic models in molecular chemistry. From among these examples, the motivating question here is: How do wind-driven waves produce ocean surface currents?The paper first summarises the geometric mechanics approach for deriving hybrid models of multiscale, multiphysics motions in ideal fluid dynamics. It then illustrates this approach for WMFI in the examples of 3D WKB waves and 2D wave amplitudes governed by the nonlinear Schrödinger (NLS) equation propagating in the frame of motion of an ideal incompressible inhomogeneous Euler fluid flow. The results for these examples tell us that the mean flow in WMFI does not create waves, although it does transport the waves. However, feedback in the opposite direction is possible, since the 3D WKB and 2D NLS wave dynamics discussed here do in fact create circulatory mean flow.

3D Information Guided Motion Transfer via Sequential Image Based Human Model Refinement and Face-Attention GAN

Image and video based human motions can be regarded as the deformation processes of person appearances, so motion transfer is usually treated as a pose guided image generation task and implemented in the 2D image plane. However, the 2D plane image generation lacks guidance of the original 3D motion information, which results in blur and shape distortions of the generated motion images. Therefore, we propose to simulate the generation process of real motion images by projecting the 3D human models, which are reconstructed from the training motion images and driven with target poses, into the 2D plane. We then take the 2D projections as the pose representations and input them into the generation model as they naturally inherit the 3D information from the original motions. Considering the unreliability on the invisible surface of the single image based human model reconstruction, we propose a sequential image based human model refinement module which exploits the complementary information between adjacent motion frames to refine the 3D human model. Furthermore, we propose a face-attention GAN model to conduct the final motion transfer, in which we use the Gaussian distribution to match the elliptical face region and design a face enhancement loss function since the faces in the generated motion images influence the performances very much. The generated motion images with reliable depth information, accurate shapes and clear faces demonstrate the effectiveness of the proposed method.

Self-Supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences

Self-supervised learning has demonstrated remarkable capability in representation learning for skeleton-based action recognition. Existing methods mainly focus on applying global data augmentation to generate different views of the skeleton sequence for contrastive learning. However, due to the rich action clues in the skeleton sequences, existing methods may only take a global perspective to learn to discriminate different skeletons without thoroughly leveraging the local relationship between different skeleton joints and video frames, which is essential for real-world applications. In this work, we propose a Partial Spatio-Temporal Learning (PSTL) framework to exploit the local relationship from a partial skeleton sequences built by a unique spatio-temporal masking strategy. Specifically, we construct a negative-sample-free triplet steam structure that is composed of an anchor stream without any masking, a spatial masking stream with Central Spatial Masking (CSM), and a temporal masking stream with Motion Attention Temporal Masking (MATM). The feature cross-correlation matrix is measured between the anchor stream and the other two masking streams, respectively. (1) Central Spatial Masking discards selected joints from the feature calculation process, where the joints with a higher degree of centrality have a higher possibility of being selected. (2) Motion Attention Temporal Masking leverages the motion of action and remove frames that move faster with a higher possibility. Our method achieves state-of-the-art performance on NTURGB+D 60, NTURGB+D 120 and PKU-MMD under various downstream tasks. Furthermore, to simulate the real-world scenarios, a practical evaluation is performed where some skeleton joints are lost in downstream tasks.In contrast to previous methods that suffer from large performance drops, our PSTL can still achieve remarkable results under this challenging setting, validating the robustness of our method.

A New Surveillance and Security Alert System Based on Real-Time Motion Detection

Due to the frequent and widespread instances of burglary, effective and accurate intrusion detection with an alarm system is now necessary. In this study, the proposed system is used for secret and vital places that need a good monitoring system based on human motion detection. The system can protect museums that contain critical historical objects and libraries with old historical books. The security alert system is also essential to the protection of banks. The system uses a two-frame difference algorithm and a simple background modeling algorithm for motion detection with the help of AForge.NET. Whenever the system detects motion, it interprets the frames of motion to determine humans by using YOLO. The security alert system activates the alerts whenever it detects human motion, like sending an email to an authorized person, triggering the LED alert and sound alert, recording a video of that place, taking a motion picture, and saving it within the databases. The system has the advantages of low cost, a user-friendly interface, and high security. The system was thoroughly tested to detect human movement in unauthorized areas. The system successfully detected human activity indoors and outdoors and activated the alarms. Also, the system could detect human movement in all four directions and alert observers directly at the time of the incident. In low light or minimum illumination, the system can determine human motion. The evaluation results showed that the system's accuracy in detecting human movement was 95%, which shows that the system is able to protect vital and secret places and can successfully monitor areas.

Multimodal dance style transfer

This paper first presents CycleDance, a novel dance style transfer system that transforms an existing motion clip in one dance style into a motion clip in another dance style while attempting to preserve the motion context of the dance. CycleDance extends existing CycleGAN architectures with multimodal transformer encoders to account for the music context. We adopt a sequence length-based curriculum learning strategy to stabilize training. Our approach captures rich and long-term intra-relations between motion frames, which is a common challenge in motion transfer and synthesis work. Building upon CycleDance, we further propose StarDance, which enables many-to-many mappings across different styles using a single generator network. Additionally, we introduce new metrics for gauging transfer strength and content preservation in the context of dance movements. To evaluate the performance of our approach, we perform an extensive ablation study and a human study with 30 participants, each with 5 or more years of dance experience. Our experimental results show that our approach can generate realistic movements with the target style, outperforming the baseline CycleGAN and its variants on naturalness, transfer strength, and content preservation. Our proposed approach has potential applications in choreography, gaming, animation, and tool development for artistic and scientific innovations in the field of dance.

Research on effects of rolling motion on a pressurizer surge line based on the fluid-solid-thermal coupling method

To investigate the effects of rolling motion on the thermal stress and deformation caused by thermal stratification of a pressurizer surge line in floating nuclear power plants (FNPP), the finite element simulation platform ANSYS Workbench 19.0 is utilized to conduct the fluid–solid-thermal coupling transient analysis of the surge line under different rolling motion conditions. The Large Eddy Simulation (LES) approach and the Frame Motion model are used in Computational Fluid Dynamics (CFD) analysis. By comparing different calculated results, it is found that rolling motion can significantly improve thermal stratification and reduce the peak thermal stress in some positions of the surge line, but may also result in periodic oscillations in temperature and thermal stress. The effects of the rolling period and the maximum rolling angle on the oscillation amplitude of temperature and thermal stress are investigated. This work provides a reference for evaluating pipeline fatigue damage and the structural integrity of the pressurizer surge line in FNPP under actual ocean conditions.

Dynamic machine vision with retinomorphic photomemristor-reservoir computing

Dynamic machine vision requires recognizing the past and predicting the future of a moving object based on present vision. Current machine vision systems accomplish this by processing numerous image frames or using complex algorithms. Here, we report motion recognition and prediction in recurrent photomemristor networks. In our system, a retinomorphic photomemristor array, working as dynamic vision reservoir, embeds past motion frames as hidden states into the present frame through inherent dynamic memory. The informative present frame facilitates accurate recognition of past and prediction of future motions with machine learning algorithms. This in-sensor motion processing capability eliminates redundant data flows and promotes real-time perception of moving objects for dynamic machine vision.

Considerations on the Relativity of Quantum Irrealism

The study of quantum resources in the relativistic limit has attracted attention over the last couple of decades, mostly due to the observation that the spin-momentum entanglement is not Lorentz covariant. In this work, we take the investigations of relativistic quantum information a step further, bringing the foundational question of realism to the discussion. In particular, we examine whether Lorentz boosts can affect quantum irrealism-an instance related to the violations imposed by quantum mechanics onto a certain notion of realism. To this end, we adopt as a theoretical platform a model of a relativistic particle traveling through a Mach-Zehnder interferometer. We then compare the quantum irrealism assessed from two different inertial frames in relative motion. In consonance with recent findings in the context of quantum reference frames, our results suggest that the notion of physical realism is not absolute.

ROTATION MINIMIZING SPHERICAL MOTIONS AND HELICES

In this study we give the structure of the motion RMM with the spherical curve orbit by using rotation minimizing frames (RMF). Further, quaternionic helices and their characterizations of being CCR curve are given with the help of unit quaternions which are related spherical frame motion.

Continuous frame motion sensitive self-supervised collaborative network for video representation learning

Motion, as a feature of video that changes in temporal sequences, is crucial to visual understanding. The powerful video representation and extraction models are typically able to focus attention on motion features in challenging dynamic environments to complete more complex video understanding tasks. However, previous approaches discriminate mainly based on similar features in the spatial or temporal domain, ignoring the interdependence of consecutive video frames. In this paper, we propose the motion sensitive self-supervised collaborative network, a video representation learning framework that exploits a pretext task to assist feature comparison and strengthen the spatiotemporal discrimination power of the model. Specifically, we first propose the motion-aware module, which extracts consecutive motion features from the spatial regions by frame difference. The global–local contrastive module is then introduced, with context and enhanced video snippets being defined as appropriate positive samples for a broader feature similarity comparison. Finally, we introduce the snippet operation prediction module, which further assists contrastive learning to obtain more reliable global semantics by sensing changes in continuous frame features. Experimental results demonstrate that our work can effectively extract robust motion features and achieve competitive performance compared with other state-of-the-art self-supervised methods on downstream action recognition and video retrieval tasks.

Hierarchical Motion Excitation Network for Few-Shot Video Recognition

Most of the existing deep learning algorithms are supervised learning and rely on a tremendous number of manually labeled samples. However, in most domains, due to the scarcity of samples or the excessive cost of labeling, it would be impracticable to provide numerous labeled training samples to the network. In this paper, a few-shot video classification network termed Hierarchical Motion Excitation Network (HME-Net) is proposed from the perspective of accumulated feature-level motion information. An HME module composed of Motion Excitation (ME) and Interval Frame Motion Excitation (IFME) is designed to extract feature-level motion patterns from adjacent frames and interval frames. The HME module can discover and enhance the feature-level motion-sensitive information in the original features. The accumulative time window is expanded to four frames in a hierarchical manner, which achieves the purpose of increasing the receptive field. After extensive experimentation, HME-Net is demonstrated to be able to consistently outperform the existing few-shot video classification models. On the UCF101 and HMDB51 datasets, our method is established as a new state-of-the-art technique for the few-shot settings of five-way three-shot and five-way five-shot video recognition.

Lightweight Deep Neural Network Embedded with Stochastic Variational Inference Loss Function for Fast Detection of Human Postures.

Fusing object detection techniques and stochastic variational inference, we proposed a new scheme for lightweight neural network models, which could simultaneously reduce model sizes and raise the inference speed. This technique was then applied in fast human posture identification. The integer-arithmetic-only algorithm and the feature pyramid network were adopted to reduce the computational complexity in training and to capture features of small objects, respectively. Features of sequential human motion frames (i.e., the centroid coordinates of bounding boxes) were extracted by the self-attention mechanism. With the techniques of Bayesian neural network and stochastic variational inference, human postures could be promptly classified by fast resolving of the Gaussian mixture model for human posture classification. The model took instant centroid features as inputs and indicated possible human postures in the probabilistic maps. Our model had better overall performance than the baseline model ResNet in mean average precision (32.5 vs. 34.6), inference speed (27 vs. 48 milliseconds), and model size (46.2 vs. 227.8 MB). The model could also alert a suspected human falling event about 0.66 s in advance.

HGRBOL2: Human gait recognition for biometric application using Bayesian optimization and extreme learning machine

The goal of gait recognition is to identify a person from a distance based on their walking style using a visual camera. However, the covariates such as a walk with carrying a bag and a change in clothes impact the recognition accuracy. This paper proposed a framework for human gait recognition based on deep learning and Bayesian optimization. The proposed framework includes both sequential and parallel steps. In the first step, optical flow-based motion regions are extracted and utilized to train the fine-tuned EfficentNet-B0 deep model. Simultaneously, we proposed a video frame enhancement technique. The same model is also separately trained on enhanced video frames. Instead of selecting static hyperparameters during the training process, Bayesian Optimization is employed in this work. Following that, two models have obtained: the motion frames model and the original frames model. Features are extracted from both models and fused using a proposed parallel approach named-Sq-Parallel Fusion (SqPF). The Tiger optimization algorithm was then improved and named Entropy controlled Tiger optimization (EVcTO). The best features are finally classified using an extreme learning machine (ELM) classifier. The experiment was carried out on two publicly available datasets, CASIA B and CASIA C, and yielded average accuracy of 92.04 and 94.97%, respectively. The proposed framework outperforms other deep learning-based networks in terms of accuracy.

Fundamentally different representations of color and motion revealed by individual differences in perceptual scaling

The coordinate frames for color and motion are often defined by three dimensions (e.g., responses from the three types of human cone photoreceptors for color and the three dimensions of space for motion). Does this common dimensionality lead to similar perceptual representations? Here we show that the organizational principles for the representation of hue and motion direction are instead profoundly different. We compared observers' judgments of hue and motion direction using functionally equivalent stimulus metrics, behavioral tasks, and computational analyses, and used the pattern of individual differences to decode the underlying representational structure for these features. Hue judgments were assessed using a standard "hue-scaling" task (i.e., judging the proportion of red/green and blue/yellow in each hue). Motion judgments were measured using a "motion-scaling" task (i.e., judging the proportion of left/right and up/down motion in moving dots). Analyses of the interobserver variability in hue scaling revealed multiple independent factors limited to different local regions of color space. This is inconsistent with the influences across a broad range of hues predicted by conventional color-opponent models. In contrast, variations in motion scaling were characterized by more global factors plausibly related to variation in the relative weightings of the cardinal spatial axes. These results suggest that although the coordinate frames for specifying color and motion share a common dimensional structure, the perceptual coding principles for hue and motion direction are distinct. These differences might reflect a distinction between the computational strategies required for the visual analysis of spatial vs. nonspatial attributes of the world.

Biomac3D: 2D-to-3D Human Pose Analysis Model for Tele-Rehabilitation Based on Pareto Optimized Deep-Learning Architecture

The research introduces a unique deep-learning-based technique for remote rehabilitative analysis of image-captured human movements and postures. We present a ploninomial Pareto-optimized deep-learning architecture for processing inverse kinematics for sorting out and rearranging human skeleton joints generated by RGB-based two-dimensional (2D) skeleton recognition algorithms, with the goal of producing a full 3D model as a final result. The suggested method extracts the entire humanoid character motion curve, which is then connected to a three-dimensional (3D) mesh for real-time preview. Our method maintains high joint mapping accuracy with smooth motion frames while ensuring anthropometric regularity, producing a mean average precision (mAP) of 0.950 for the task of predicting the joint position of a single subject. Furthermore, the suggested system, trained on the MoVi dataset, enables a seamless evaluation of posture in a 3D environment, allowing participants to be examined from numerous perspectives using a single recorded camera feed. The results of evaluation on our own self-collected dataset of human posture videos and cross-validation on the benchmark MPII and KIMORE datasets are presented.

A Deep Learning Method for Motion Artifact Correction in Intravascular Photoacoustic Image Sequence.

In vivo application of intravascular photoacoustic (IVPA) imaging for coronary arteries is hampered by motion artifacts associated with the cardiac cycle. Gating is a common strategy to mitigate motion artifacts. However, a large amount of diagnostically valuable information might be lost due to one frame per cycle. In this work, we present a deep learning-based method for directly correcting motion artifacts in non-gated IVPA pullback sequences. The raw signal frames are classified into dynamic and static frames by clustering. Then, a neural network named Motion Artifact Correction (MAC)-Net is designed to correct motion in dynamic frames. Given the lack of the ground truth information on the underlying dynamics of coronary arteries, we trained and tested the network using a computer-generated dataset. Based on the results, it has been observed that the trained network can directly correct motion in successive frames while preserving the original structures without discarding any frames. The improvement in the visual effect of the longitudinal view has been demonstrated based on quantitative evaluation of the inter-frame dissimilarity. The comparison results validated the motion-suppression ability of our method comparable to gating and image registration-based non-learning methods, while maintaining the integrity of the pullbacks without image preprocessing. Experimental results from in vivo intravascular ultrasound and optical coherence tomography pullbacks validated the feasibility of our method in the in vivo intracoronary imaging scenario.