Global Motion Patterns Research Articles

Stochastic video normality network for abnormal event detection in surveillance videos

Video Anomaly Detection (VAD) aims to automatically identify unexpected spatial–temporal patterns to detect abnormal events in surveillance videos. Existing unsupervised VAD methods either use a single proxy task to represent the prototypical normal patterns, or split the video into static and dynamic parts and learn spatial and temporal normality separately, which cannot model the inherent uncertainty of motion. In this regard, we propose for the first time to learn video normality via stochastic motion representation. Specifically, the proposed Stochastic Video Normality (SVN) network learns the prototypical local appearance patterns via deterministic multi-task learning and global motion patterns in a non-deterministic manner. The stochastic representation module uses recurrent networks to model historical motion as a conditional Gaussian distribution and infers motion explicitly with the learned prior. Besides, a mask auto-encoder is introduced to explore the relationship between appearance and motion, boosting the model to learn spatial–temporal normality. Experimental results show that our proposed SVN network performs comparably to the state-of-the-art methods. Extensive analysis demonstrates the effectiveness of multi-task learning-based appearance representation and stochastic motion representation for unsupervised video anomaly detection.

Motion and form coherence processing in individuals with cerebral visual impairment.

Using a visual psychophysical paradigm, we sought to assess motion and form coherence thresholds as indices of dorsal and ventral visual stream processing respectively, in individuals with cerebral visual impairment (CVI). We also explored potential associations between psychophysical assessments and brain lesion severity in CVI. Twenty individuals previously diagnosed with CVI (mean age = 17 years 11 months [SD 5 years 10 months]; mean Verbal IQ = 86.42 [SD 35.85]) and 30 individuals with neurotypical development (mean age = 20 years 1 month [SD 3 years 8 months]; mean Verbal IQ = 110.05 [SD 19.34]) participated in the study. In this two-group comparison, cross-sectional study design, global motion, and form pattern coherence thresholds were assessed using a computerized, generalizable, self-administrable, and response-adaptive psychophysical paradigm called FInD (Foraging Interactive D-prime). Consistent with dorsal stream dysfunction, mean global motion (but not form) coherence thresholds were significantly higher in individuals with CVI compared to controls. No statistically significant association was found between coherence thresholds and lesion severity. These results suggest that the objective assessment of motion and form coherence threshold sensitivities using this psychophysical paradigm may be useful in helping to characterize perceptual deficits and the complex clinical profile of CVI. In participants with cerebral visual impairment (CVI), motion (but not form) coherence thresholds were significantly higher compared to controls. These psychophysical results support the notion of dorsal stream dysfunction in CVI.

Recursive Least Squares Based Refinement Network for Vehicle Trajectory Prediction

Trajectory prediction of surrounding objects plays a pivotal role in the field of autonomous driving vehicles. In the current rollout process, it suffers from an accumulation of errors, which has a negative impact on prediction accuracy. This paper proposes a parametric-learning recursive least-squares (RLS) method integrated with an encoder–decoder framework for trajectory prediction, named the recursive least-squares-based refinement network (RRN). Through the generation of several anchors in the future trajectory, RRN can capture both local and global motion patterns. We conducted experiments on the prevalent NGSIM and INTERACTION datasets, which contain various scenarios such as highways, intersections and roundabouts. The promising results indicate that RRN could improve the performance of the rollout trajectory prediction effectively.

Behaviour Detection and Recognition of College Basketball Players Based on Multimodal Sequence Matching and Deep Neural Networks.

This study fuses multimodal sequence matching with a deep neural network algorithm for college basketball player behavior detection and recognition to conduct in-depth research and analysis, analyzing the basic components of basketball technical action videos by studying the practical application of technical actions in professional games and teaching videos from self-published authors of short video platforms. The characteristics of the dataset are also analyzed through literature research related to the basketball action dataset. On the established basketball technical action dataset, combined with the SSD target detection algorithm, the video images with cropping of human motion regions to reduce the size of image frames, generating a basketball technical action dataset based on cropped frames, reduces the amount of network training and improves the efficiency of subsequent action recognition training. In this study, by analyzing the characteristics of basic camera motion, a univariate global motion model is proposed to introduce a quadratic term to accurately express the shaking transformation, while the horizontal and vertical motion are independently represented to reduce the model complexity. Comparative experimental results show that the proposed model achieves a good balance between complexity and accuracy of global motion representation. It is suitable for global motion modeling in behavior recognition applications, laying the foundation for global and local motion estimation. On this basis, the visual feature change pattern of the key area of the scene (basketball area) is combined with the group behavior recognition based on motion patterns and the success-failure classification based on key visual information to achieve basketball semantic event recognition. The experimental results at NCAA show that the fusion of global and local motion patterns can effectively improve group behavior recognition performance. The semantic event recognition algorithm combining motion patterns and video key visual information achieves the best performance.

Video Analysis and System Construction of Basketball Game by Lightweight Deep Learning under the Internet of Things.

With the explosive growth of sports video data on the internet platform, how to scientifically manage this information has become a major challenge in the current big data era. In this context, a new lightweight player segmentation algorithm is proposed to realize the automatic analysis of basketball game video. Firstly, semantic events are expressed by extracting group and global motion features. A complete basketball game video is divided into three stages, and a basketball event classification method integrating global group motion patterns and domain knowledge is proposed. Secondly, a player segmentation algorithm based on lightweight deep learning is proposed to detect basketball players, segment the players, and finally extract players' spatial features based on deep learning to realize players' pose estimation. As the experimental results indicate, when a proposed 2-stage classification algorithm is used to classify the videos, the accuracy of identifying layup, the shooting, and other 2-pointers are improved by 21.26% and 6.41%, respectively. And the accuracy of average events sees an improvement of 2.74%. The results imply that the 2-stage classification based on event-occ is effective. After comparing the four methods of classifying players, it is found that there is no significant difference among these four methods about the accuracy of segmenting. Nevertheless, when judged with the time that these methods take separately, FCN-CNN (Fully Convolutional Network-Convolutional Neural Network) based on superpixels has overwhelming advantages. The event analysis method of basketball game video proposed here can realize the automatic analysis of basketball video, which is beneficial to promoting the rapid development of basketball and even sports.

Populations of local direction-selective cells encode global motion patterns generated by self-motion.

Self-motion generates visual patterns on the eye that are important for navigation. These optic flow patterns are encoded by the population of local direction–selective cells in the mouse retina, whereas in flies, local direction–selective T4/T5 cells are thought to be uniformly tuned. How complex global motion patterns can be computed downstream is unclear. We show that the population of T4/T5 cells in Drosophila encodes global motion patterns. Whereas the mouse retina encodes four types of optic flow, the fly visual system encodes six. This matches the larger number of degrees of freedom and the increased complexity of translational and rotational motion patterns during flight. The four uniformly tuned T4/T5 subtypes described previously represent a local subset of the population. Thus, a population code for global motion patterns appears to be a general coding principle of visual systems that matches local motion responses to modes of the animal’s movement.

Motion pattern-based crowd scene classification using histogram of angular deviations of trajectories

Automated crowd behaviour analysis and monitoring is a challenging task due to the unpredictable nature of the crowd within a particular scene and across different scenes. The prior knowledge of the type of scene under consideration is a crucial mid-level information, which could be utilized to develop robust crowd behaviour analysis systems. In this paper, we propose an approach to automatically detect the type of a crowded scene based on the global motion patterns of the objects within the scene. Three different types of scenes whose global motion pattern characteristics vary from uniform to non-uniform are considered in this work, namely structured, semi-structured, and unstructured scenes, respectively. To capture the global motion pattern characteristics of an input crowd scene, we first extract the motion information in the form of trajectories using a key-point tracker and then compute the average angular orientation feature of each trajectory. This paper utilizes these angular features to introduce a novel feature vector, termed as Histogram of Angular Deviations (HAD), which depicts the distribution of the pair-wise angular deviation values for each trajectory vector. Since angular deviation information is resistant to changes in scene perspectives, we consider it as a key feature for distinguishing the scene types. To evaluate the effectiveness of the proposed HAD-based feature vector in classifying the crowded scenes, we build a crowd scene classification model by training the classical machine learning algorithms on the publicly available Collective Motion Database. The experimental results demonstrate the superior crowd classification performance of the proposed approach as compared to the existing methods. In addition to this, we propose a technique based on quantizing the angular deviation values to reduce the feature dimension and subsequently introduce a novel crowd scene structuredness index to quantify the structuredness of an input crowded scene based on its HAD.

Local motion pooling is continuous, global motion perception is discrete.

Perceiving the motion of an object is thought to involve two stages: Local motion energy is measured at each point in space, and these signals are then pooled across space to build coherent global motion. There are several theories of how local-to-global pooling occurs, but they all predict that global motion perception is a continuous process, such that increasing the strength of motion energy should gradually increase the precision of perceived motion directions. We test this prediction against the alternative that global motion perception is discrete: Motion is either perceived with high precision or fails to be perceived altogether. Data from human observers provides clear evidence that, whereas pooling local motion energy is continuous, the segmentation of local signals into coherent global motion patterns is a discrete process. This result adds motion perception to the growing list of processes that exhibit evidence of all-or-none visual awareness. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Perception of Dynamic Point Light Facial Expression

Abstract This study uses point light displays both to investigate the roles of global and local motion analyses in the perception of dynamic facial expressions and to measure the information threshold for reliable recognition of emotions. We videotaped the faces of actors wearing black makeup with white dots while they dynamically produced each of 6 basic Darwin/Ekman emotional expressions. The number of point lights was varied to systematically manipulate amount of information available. For all but one of the expressions, discriminability (d′) increased approximately linearly with number of point lights, with most remaining largely discriminable with as few as only 6 point lights. This finding supports reliance on global motion patterns produced by facial muscles. However, discriminability for the happy expression was notably higher and largely unaffected by number of point lights and thus appears to rely on characteristic local motion, probably the unique upward curvature of the mouth. The findings indicate that recognition of facial expression is not a unitary process and that different expressions may be conveyed by different perceptual information, but in general, basic facial emotional expressions typically remain largely discriminable with as few as 6 dynamic point lights.

Contribution of the slow motion mechanism to global motion revealed by an MAE technique

Two different motion mechanisms have been identified with motion aftereffect (MAE). (1) A slow motion mechanism, accessed by a static MAE, is sensitive to high-spatial and low-temporal frequency; (2) a fast motion mechanism, accessed by a flicker MAE, is sensitive to low-spatial and high-temporal frequency. We examined their respective responses to global motion after adapting to a global motion pattern constructed of multiple compound Gabor patches arranged circularly. Each compound Gabor patch contained two gratings at different spatial frequencies (0.53 and 2.13 cpd) drifting in opposite directions. The participants reported the direction and duration of the MAE for a variety of global motion patterns. We discovered that static MAE durations depended on the global motion patterns, e.g., longer MAE duration to patches arranged to see rotation than to random motion (Exp 1), and increase with global motion strength (patch number in Exp 2). In contrast, flicker MAEs durations are similar across different patterns and adaptation strength. Further, the global integration occurred at the adaptation stage, rather than at the test stage (Exp 3). These results suggest that slow motion mechanism, assessed by static MAE, integrate motion signals over space while fast motion mechanisms do not, at least under the conditions used.

Development of Basketball Tactics Basic Cooperation Teaching System Based on CNN and BP Neural Network.

The traditional basketball teaching mode cannot meet the needs of students for the basic cooperation of basketball tactics. Therefore, a basic cooperation teaching system of basketball tactics based on artificial neural network is studied and designed. The system has a professional basketball game video tactical learning module. The events in the basketball game video are classified through a convolutional neural network and combined with the explanation of teachers to make the students have an intuitive understanding of the basic cooperation of basketball tactics and then design the basketball game module based on a BP neural network to provide students with an online basketball tactics training platform. Finally, the teacher scores the performance of the actual on-site training students in the basic cooperation of basketball tactics through the tactical scoring module on the system. The results show that after the introduction of global and collective motion patterns, the classification accuracy of the convolutional neural network is improved by 22.48%, which has significant optimization. The average accuracy of basketball game video event classification is 62.35%, and the accuracy of snatch event classification is improved to 95.28%. The recognition rate of the BP neural network combined with momentum gradient descent method is 75%, the number of weight adjustment is less, and the memory is small while ensuring fast running speed. Students who accept the basic basketball tactics cooperation teaching system based on the artificial neural network for basketball teaching have an overall score of 27.99 ± 2.11 points The overall score of exchange defense cooperation was 24.12 ± 2.03, which was higher than that of the control group. The above results show that the basketball tactical basic cooperation teaching system based on the artificial neural network has a good teaching effect in improving students' basketball tactical basic cooperation ability.

Fusing motion patterns and key visual information for semantic event recognition in basketball videos

Many semantic events in team sport activities e.g. basketball often involve both group activities and the outcome (score or not). Motion patterns can be an effective means to identify different activities. Global and local motions have their respective emphasis on different activities, which are difficult to capture from the optical flow due to the mixture of global and local motions. Hence it calls for a more effective way to separate the global and local motions. When it comes to the specific case for basketball game analysis, the successful score for each round can be reliably detected by the appearance variation around the basket. Based on the observations, we propose a scheme to fuse global and local motion patterns (MPs) and key visual information (KVI) for semantic event recognition in basketball videos. Firstly, an algorithm is proposed to estimate the global motions from the mixed motions based on the intrinsic property of camera adjustments. And the local motions could be obtained from the mixed and global motions. Secondly, a two-stream 3D CNN framework is utilized for group activity recognition over the separated global and local motion patterns. Thirdly, the basket is detected and its appearance features are extracted through a CNN structure. The features are utilized to predict the success or failure. Finally, the group activity recognition and success/failure prediction results are integrated using the kronecker product for event recognition. Experiments on NCAA dataset demonstrate that the proposed method obtains state-of-the-art performance.

Computational Mechanisms for Perceptual Stability using Disparity and Motion Parallax.

Walking and other forms of self-motion create global motion patterns across our eyes. With the resulting stream of visual signals, how do we perceive ourselves as moving through a stable world? Although the neural mechanisms are largely unknown, human studies (Warren and Rushton, 2009) provide strong evidence that the visual system is capable of parsing the global motion into two components: one due to self-motion and the other due to independently moving objects. In the present study, we use computational modeling to investigate potential neural mechanisms for stabilizing visual perception during self-motion that build on neurophysiology of the middle temporal (MT) and medial superior temporal (MST) areas. One such mechanism leverages direction, speed, and disparity tuning of cells in dorsal MST (MSTd) to estimate the combined motion parallax and disparity signals attributed to the observer's self-motion. Feedback from the most active MSTd cell subpopulations suppresses motion signals in MT that locally match the preference of the MSTd cell in both parallax and disparity. This mechanism combined with local surround inhibition in MT allows the model to estimate self-motion while maintaining a sparse motion representation that is compatible with perceptual stability. A key consequence is that after signals compatible with the observer's self-motion are suppressed, the direction of independently moving objects is represented in a world-relative rather than observer-relative reference frame. Our analysis explicates how temporal dynamics and joint motion parallax-disparity tuning resolve the world-relative motion of moving objects and establish perceptual stability. Together, these mechanisms capture findings on the perception of object motion during self-motion.SIGNIFICANCE STATEMENT The image integrated by our eyes as we move through our environment undergoes constant flux as trees, buildings, and other surroundings stream by us. If our view can change so radically from one moment to the next, how do we perceive a stable world? Although progress has been made in understanding how this works, little is known about the underlying brain mechanisms. We propose a computational solution whereby multiple brain areas communicate to suppress the motion attributed to our movement relative to the stationary world, which is often responsible for a large proportion of the flux across the visual field. We simulated the proposed neural mechanisms and tested model estimates using data from human perceptual studies.

Multiple stream deep learning model for human action recognition

Human action recognition is one of the most important and challenging topic in the fields of image processing. Unlike object recognition, action recognition requires motion feature modeling which contains not only spatial but also temporal information. In this paper, we use multiple models to characterize both global and local motion features. Global motion patterns are represented efficiently by the depth-based 3-channel motion history images (MHIs). Meanwhile, the local spatial and temporal patterns are extracted from the skeleton graph. The decisions of these two streams are fused. At the end, the domain knowledge, which is the object/action dependency is considered. The proposed framework is evaluated on two RGB-D datasets. The experimental results show the effectiveness of our proposed approach. The performance is comparable with the state-of-the-art.

Short-term global motion adaptation induces a compression in the subjective duration of dynamic visual events

Apparent duration can be manipulated in a local region of visual field by long-term adaptation to motion or flicker (Johnston, Arnold, & Nishida, 2006). These effects show narrow spatial tuning (Ayhan, Bruno, Nishida, & Johnston, 2009), as well as retinotopic position dependency (Bruno, Ayhan, & Johnston, 2010), supporting early locus in the visual pathway. Here, we introduce a new effect using RDK as a short-term visual adaptor and demonstrate that a brief, subsecond range adaptation induces a significant subjective duration compression (∼10%) on a subsequently presented test stimulus (RDK pattern) only for global motion patterns drifting at 50% motion coherence but not for those drifting at 0% coherence, suggesting a higher level area as a source of origin. In another set of experiments using a plaid stimulus as the adaptor and gratings as the tests, we report again a significant duration compression following a brief motion adaptation, although the effect does not seem to be consistently selective for a particular direction of the standard test relative to that of the plaid adaptor (two-dimensional motion) or its components (one-dimensional motion). Finally, we conduct an experiment using shutter glasses and find that the effects of a short-term adaptor presented monocularly to one eye transfer to the nonadapted eye, providing evidence for the interocular transfer. In a series of control experiments, we also show that the duration effects cannot be explained by adaptation-induced changes in perceived speed, perceived onset-and-offset, and attentional resource allocation. Overall, the duration compression effect requiring motion coherence in RDK, persisting in plaid stimulus, and showing interocular transfer imply explicit genuine mechanisms mediating duration effects in the higher level motion areas.

Human action recognition using bag of global and local Zernike moment features

Human action recognition is a fundamental and challenging building block for many computer vision applications. It has been included in many applications such as: video surveillance, human computer interaction and multimedia retrieval systems. Various approaches have been proposed to solve human action recognition problem. Among others, moment-based methods considered as one of the most simple and successful approach. However, moment-based methods take into consideration only global features while neglect the discriminative properties of local features. In this paper, we propose a new efficient method which combine both Global and Local Zernike Moment (GLZM) features based on Bag-of-Features (BoF) technique. Since using only global features are not sufficient to discriminate similar actions like running, walking and jogging, augmenting these features with localized features helps to improve the recognition accuracy. The proposed method first calculate local temporal Motion Energy Images (MEI) by accumulating frame differences of short time consecutive frames. Then, global and local features are calculated using Zernike moments with different polynomial orders to represent global and local motion patterns respectively. Global features are calculated from the whole region of the human performing action while local features focused on localized regions of the human in order to represent local motion information. Both local and global features are preprocessed using whitening transformation, then bag-of-features algorithm is employed to combine those pool of features and represent each action using new GLZM feature descriptor. Finally, we use multi-class Support Vector Machine (SVM) classifier to recognize human actions. In order to validate the proposed method, we perform a set of experiments using three publicly available datasets: Weizmann, KTH and UCF sports action. Experimental results using leave-one-out strategy show that proposed method achieves promising results compared with other state-of-the-art methods.

Symmetry considerations elucidate the roles of global shape and local interactions in the equilibrium fluctuations and cooperativity of protein assemblies

Shape had been intuitively recognized to play a dominant role in determining the global motion patterns of bio-molecular assemblies. However, it is not clear exactly how shape determines the motion patterns. What about the local interactions that hold a structure together to a certain shape? The contributions of global shape and local interactions usually mix together and are difficult to tease part. In this work, we use symmetry to elucidate the distinct roles of global shape and local interactions in protein dynamics. Symmetric complexes provide an ideal platform for this task since in them the effects of local interactions and global shape are separable, allowing their distinct roles to be identified. Our key findings based on symmetric assemblies are: (i) the motion patterns of each subunit are determined primarily by intra-subunit interactions (IRSi), and secondarily by inter-subunit interactions (IESi); (ii) the motion patterns of the whole assembly are fully dictated by the global symmetry/shape and have nothing to do with local iESi or IRSi. This is followed by a discussion on how the findings may be generalized to complexes in any shape, with or without symmetry.

Efficient PatchMatch-Based Synthesis for Cartoon Animation

Automating the production of 2D hand-drawn animations is a significant and interesting component in computer graphics and vision. However, traditional methods in animation production pipeline always use physically or geometrically based models which are consuming due to complicated and massive computations, reducing their practicability. In this paper, we propose an efficient data-driven approach to create hand-drawn animations in an automatic manner. The key idea is to employ a correspondence match-based random search process to extract the geometry samples and the global motion pattern in an input animation sequence and then to generate a new output sequence through a coarse-to-fine sample-based synthesis algorithm. Our experiments demonstrate that our method achieves good results with high quality and performance, producing a range of artistic effects that previously required disparate and professional techniques.

Ontology Based Global and Collective Motion Patterns for Event Classification in Basketball Videos

In multi-person videos, especially team sport videos, a semantic event is usually represented as a confrontation between two teams of players, which can be represented as collective motion. In broadcast basketball videos, specific camera motions are used to present specific events. Therefore, a semantic event in broadcast basketball videos is closely related to both the global motion (camera motion) and the collective motion. A semantic event in basketball videos can be generally divided into three stages: pre-event, event occurrence (event-occ), and post-event. In this paper, we propose an ontology-based global and collective motion pattern (On_GCMP) algorithm for basketball event classification. First, a two-stage GCMP based event classification scheme is proposed. The GCMP is extracted using optical flow. The two-stage scheme progressively combines a five-class event classification algorithm on event-occs and a two-class event classification algorithm on pre-events. Both algorithms utilize sequential convolutional neural networks (CNNs) and long short-term memory (LSTM) networks to extract the spatial and temporal features of GCMP for event classification. Second, we utilize post-event segments to predict success/failure using deep features of images in the video frames (RGB_DF_VF) based algorithms. Finally the event classification results and success/failure classification results are integrated to obtain the final results. To evaluate the proposed scheme, we collected a new dataset called NCAA+, which is automatically obtained from the NCAA dataset by extending the fixed length of video clips forward and backward of the corresponding semantic events. The experimental results demonstrate that the proposed scheme achieves the mean average precision of 58.10% on NCAA+. It is higher by 6.50% than state-of-the-art on NCAA.

Characterization of Biological Motion Using Motion Sensing Superpixels.

Precise spatiotemporal regulation is the foundation for the healthy development and maintenance of living organisms. All cells must correctly execute their function in the right place at the right time. Cellular motion is thus an important dynamic readout of signaling in key disease-relevant molecular pathways. However despite the rapid advancement of imaging technology, a comprehensive quantitative description of motion imaged under different imaging modalities at all spatiotemporal scales; molecular, cellular and tissue-level is still lacking. Generally, cells move either 'individually' or 'collectively' as a group with nearby cells. Current computational tools specifically focus on one or the other regime, limiting their general applicability. To address this, we recently developed and reported a new computational framework, Motion Sensing Superpixels (MOSES). Incorporating the individual advantages of single cell trackers for individual cell and particle image velocimetry (PIV) for collective cell motion analyses, MOSES enables 'mesoscale' analysis of both single-cell and collective motion over arbitrarily long times. At the same time, MOSES readily complements existing single-cell tracking workflows with additional characterization of global motion patterns and interaction analysis between cells and also operates directly on PIV extracted motion fields to yield rich motion trajectories analogous for single-cell tracks suitable for high-throughput motion phenotyping. This protocol provides a step-by-step practical guide for those interested in applying MOSES to their own datasets. The protocol highlights the salient features of a MOSES analysis and demonstrates the ease-of-use and wide applicability of MOSES to biological imaging through demo experimental analyses with ready-to-use code snippets of four datasets from different microscope modalities; phase-contrast, fluorescent, lightsheet and intra-vital microscopy. In addition we discuss critical points of consideration in the analysis.