Motion Cues Research Articles

Accurate Object Tracking by Utilizing Diverse Prior Information

Siamese trackers draw continuous attention in the object tracking community due to their proper balance between performance and inference speed. Despite that, it remains unclear how to effectively exploit the target appearance cues and motion cues involved in videos to improve trackers’ performance. To address this problem, we develop a Siamese network with diverse prior information integrated, namely DPINet, by extending two novel blocks to a powerful anchor-free Siamese network. First, we design a channel- and space-aware feature enhancement (CSE) block to highlight target-specific feature weights in two aspects (channel and spatial dimensions). It is devoted to making full use of the target cues in the initial frame by considering them as guidances, in which way target-related representation in feature maps can be improved. It also facilitates the interplay between two input branches. Second, we advance a cross-correlation block with multi-dimensional information fusion (MDI-XCorr). In this block, target motion cues within adjacent frames can be mined and treated as supervisions to refine the response map in the current frame during inference. Hence, both tracking quality and stabilization can be enhanced. Evaluations on four popular benchmarks are conducted, showing that DPINet achieves 0.702 (AUC), 0.474 (EAO), 0.336 (EAO), 0.613 (AO), and 0.527 (AUC) on OTB100, VOT2018, VOT2019, GOT-10k, and LaSOT, respectively.

Animacy perception from motion cues: Cognitive and neural mechanisms

Animacy perception from motion cues: Cognitive and neural mechanisms

Adaptive Motion Cueing Algorithm Using Optimized Fuzzy Control System for Motion Simulators

The Motion Cueing Algorithm (MCA) is the main unit of motion simulators responsible for transforming the vehicle motions to generate driving motion sensation for the simulator users within the motion simulator's physical workspace through washout filters. In this study, we design and provide a new framework by developing a set of novel washout filters using optimized fuzzy control systems to solve the drawbacks associated with the existing optimal MCAs of the motion simulators. These drawbacks include constant washout filter parameters, inefficient simulator movement in the workspace, lack of considering the simulator's physical constraints and driver sensation-related factors, and sub-optimal settings which causes false motion cues and subsequently simulator sickness. As a pioneering framework, the fuzzy logic controllers generate restitution signals for the optimal washout filters based on the motion sensation error between the real vehicle and simulator divers as well as the platform's position in the workspace aiming to correct the filtered signals, reduce error, generate realistic motions, and increase motion fidelity online. The GA is applied to optimize the fuzzy logic membership functions and control rules. The optimization procedure takes into account a number of important factors that are normally ignored in the current MCAs including the sensed motion error between the drivers of simulator and real car; the platform physical constraints; the motion threshold limiter effects; and signal shape following. The new adaptive MCA is designed, implemented and tested using the MATLAB/Simulink. The simulation results confirm the effectiveness of the developed MCA in maximizing motion fidelity and the reference sensation shape following, and enhancing the workspace usage of the simulator.

Hierarchical Graph Pattern Understanding for Zero-Shot Video Object Segmentation.

The optical flow guidance strategy is ideal for obtaining motion information of objects in the video. It is widely utilized in video segmentation tasks. However, existing optical flow-based methods have a significant dependency on optical flow, which results in poor performance when the optical flow estimation fails for a particular scene. The temporal consistency provided by the optical flow could be effectively supplemented by modeling in a structural form. This paper proposes a new hierarchical graph neural network (GNN) architecture, dubbed hierarchical graph pattern understanding (HGPU), for zero-shot video object segmentation (ZS-VOS). Inspired by the strong ability of GNNs in capturing structural relations, HGPU innovatively leverages motion cues (i.e., optical flow) to enhance the high-order representations from the neighbors of target frames. Specifically, a hierarchical graph pattern encoder with message aggregation is introduced to acquire different levels of motion and appearance features in a sequential manner. Furthermore, a decoder is designed for hierarchically parsing and understanding the transformed multi-modal contexts to achieve more accurate and robust results. HGPU achieves state-of-the-art performance on four publicly available benchmarks (DAVIS-16, YouTube-Objects, Long-Videos and DAVIS-17). Code and pre-trained model can be found at https://github.com/NUST-Machine-Intelligence-Laboratory/HGPU.

Blind VQA on 360° Video via Progressively Learning from Pixels, Frames and Video.

Blind visual quality assessment (BVQA) on 360° video plays a key role in optimizing immersive multimedia systems. When assessing the quality of 360° video, human tends to perceive its quality degradation from the viewport-based spatial distortion of each spherical frame to motion artifact across adjacent frames, ending with the video-level quality score, i.e., a progressive quality assessment paradigm. However, the existing BVQA approaches for 360° video neglect this paradigm. In this paper, we take into account the progressive paradigm of human perception towards spherical video quality, and thus propose a novel BVQA approach (namely ProVQA) for 360° video via progressively learning from pixels, frames and video. Corresponding to the progressive learning of pixels, frames and video, three sub-nets are designed in our ProVQA approach, i.e., the spherical perception aware quality prediction (SPAQ), motion perception aware quality prediction (MPAQ) and multi-frame temporal non-local (MFTN) sub-nets. The SPAQ sub-net first models the spatial quality degradation based on spherical perception mechanism of human. Then, by exploiting motion cues across adjacent frames, the MPAQ sub-net properly incorporates motion contextual information for quality assessment on 360° video. Finally, the MFTN sub-net aggregates multi-frame quality degradation to yield the final quality score, via exploring long-term quality correlation from multiple frames. The experiments validate that our approach significantly advances the state-of-the-art BVQA performance on 360° video over two datasets, the code of which has been public in https://github.com/yanglixiaoshen/ProVQA.

Hybrid Motion Representation Learning for Prediction From Raw Sensor Data

Motion prediction from raw LiDAR sensor data has drawn increasing attention and led to a surge of studies following two main paradigms. One paradigm is global motion paradigm, which simultaneously detects objects from point clouds and predicts the trajectories of each object in the future. The other paradigm is local motion paradigm, which directly performs dense motion prediction pointwisely. We observe that global motion prediction can benefit from local motion representation, since it contains rich local displacement contexts that are not explicitly exploited in global motion prediction. Correspondingly, local motion prediction can benefit from global motion representation, since it provides object contexts to improve prediction consistency inside an object. However, the complement of these two motion representations has not fully explored in the literature. To this end, we propose Hybrid Motion Representation Learning (HyMo), a unified framework to address the problem of motion prediction by making the best of both global and local motion cues. We have conducted extensive experiments on nuScenes dataset. The experimental results demonstrate that the learned hybrid motion representation achieves state-of-the-art performance on both global and local motion prediction tasks.

Weakly Supervised Regional and Temporal Learning for Facial Action Unit Recognition

Automatic facial action unit (AU) recognition is a challenging task due to the scarcity of manual annotations. To alleviate this problem, a large amount of efforts has been dedicated to exploiting various weakly supervised methods which leverage numerous unlabeled data. However, many aspects with regard to some unique properties of AUs, such as the regional and relational characteristics, are not sufficiently explored in previous works. Motivated by this, we take the AU properties into consideration and propose two auxiliary AU related tasks to bridge the gap between limited annotations and the model performance in a self-supervised manner via the unlabeled data. Specifically, to enhance the discrimination of regional features with AU relation embedding, we design a task of RoI inpainting to recover the randomly cropped AU patches. Meanwhile, a single image based optical flow estimation task is proposed to leverage the dynamic change of facial muscles and encode the motion information into the global feature representation. Based on these two self-supervised auxiliary tasks, local features, mutual relation and motion cues of AUs are better captured in the backbone network. Furthermore, by incorporating semi-supervised learning, we propose an end-to-end trainable framework named weakly supervised regional and temporal learning (WSRTL) for AU recognition. Extensive experiments on BP4D and DISFA demonstrate the superiority of our method and new state-of-the-art performances are achieved.

Problem of motion cueing along linear degrees of freedom on flight simulators

On the basis of Gibson's perception theory, a set of characteristic attributes of human motion perception is determined. On the basis of the system approach, the mathematical statement of the motion cueing problem along linear degrees of freedom on flight simulators of non-maneuvering aircraft is substantiated. This statement guarantees a motion cueing as close as possible to the real set of characteristic attributes of the motion cueing: nature, beginning time, direction, duration, and intensity of perception.

The Impact of Physical Motion Cues on Driver Braking Performance: A Clinical Study Using Driving Simulator and Eye Tracker.

Driving simulators are increasingly being incorporated by driving schools into a training process for a variety of vehicles. The motion platform is a major component integrated into simulators to enhance the sense of presence and fidelity of the driving simulator. However, less effort has been devoted to assessing the motion cues feedback on trainee performance in simulators. To address this gap, we thoroughly study the impact of motion cues on braking at a target point as an elementary behavior that reflects the overall driver's performance. In this paper, we use an eye-tracking device to evaluate driver behavior in addition to evaluating data from a driving simulator and considering participants' feedback. Furthermore, we compare the effect of different motion levels ("No motion", "Mild motion", and "Full motion") in two road scenarios: with and without the pre-braking warning signs with the speed feedback given by the speedometer. The results showed that a full level of motion cues had a positive effect on braking smoothness and gaze fixation on the track. In particular, the presence of full motion cues helped the participants to gradually decelerate from 5 to 0 ms-1 in the last 240 m before the stop line in both scenarios, without and with warning signs, compared to the hardest braking from 25 to 0 ms-1 produced under the no motion cues conditions. Moreover, the results showed that a combination of the mild motion conditions and warning signs led to an underestimation of the actual speed and a greater fixation of the gaze on the speedometer. Questionnaire data revealed that 95% of the participants did not suffer from motion sickness symptoms, yet participants' preferences did not indicate that they were aware of the impact of simulator conditions on their driving behavior.

Disentangling Object Category Representations Driven by Dynamic and Static Visual Input.

Humans can label and categorize objects in a visual scene with high accuracy and speed, a capacity well characterized with studies using static images. However, motion is another cue that could be used by the visual system to classify objects. To determine how motion-defined object category information is processed by the brain in the absence of luminance-defined form information, we created a novel stimulus set of "object kinematograms" to isolate motion-defined signals from other sources of visual information. Object kinematograms were generated by extracting motion information from videos of 6 object categories and applying the motion to limited-lifetime random dot patterns. Using functional magnetic resonance imaging (fMRI) (n = 15, 40% women), we investigated whether category information from the object kinematograms could be decoded within the occipitotemporal and parietal cortex and evaluated whether the information overlapped with category responses to static images from the original videos. We decoded object category for both stimulus formats in all higher-order regions of interest (ROIs). More posterior occipitotemporal and ventral regions showed higher accuracy in the static condition, while more anterior occipitotemporal and dorsal regions showed higher accuracy in the dynamic condition. Further, decoding across the two stimulus formats was possible in all regions. These results demonstrate that motion cues can elicit widespread and robust category responses on par with those elicited by static luminance cues, even in ventral regions of visual cortex that have traditionally been associated with primarily image-defined form processing.SIGNIFICANCE STATEMENT Much research on visual object recognition has focused on recognizing objects in static images. However, motion is a rich source of information that humans might also use to categorize objects. Here, we present the first study to compare neural representations of several animate and inanimate objects when category information is presented in two formats: static cues or isolated dynamic motion cues. Our study shows that, while higher-order brain regions differentially process object categories depending on format, they also contain robust, abstract category representations that generalize across format. These results expand our previous understanding of motion-derived animate and inanimate object category processing and provide useful tools for future research on object category processing driven by multiple sources of visual information.

Social Inference from Relational Visual Information

Humans easily make social evaluations from visual input, such as recognizing a social interaction or deciding who is a friend and who is a foe. Prior efforts to model this ability have suggested that visual features and motion cues alone cannot account for human performance in social tasks, like distinguishing between helping and hindering interactions. On the other hand, generative Bayesian inference models, which make predictions based on simulations of agents’ social or physical goals, accurately predict human judgments, but are computationally very expensive and often impossible to implement in natural visual stimuli. Inspired by developmental work, we hypothesize that introducing inductive biases in visual models would allow them to make more human-like social judgments. Specifically, in this study, we investigate if relational representations of visual stimuli using graph neural networks can predict human social interaction judgments. We use the PHASE dataset, consisting of 2D animations of two agents and two objects resembling real-life social interactions, rated as friendly, neutral, or adversarial. We propose a graph neural networks (GNN) based architecture, that takes in graph representations of a video and predicts the relationship between the agents. We collected human ratings for each of the 400 videos and found that our GNN model aligns with human judgments significantly better than a baseline visual model with the same visual/motion information but without the graph structure (79% vs 62% prediction of human judgements). Intriguingly, explicitly adding relational information to the baseline model does not improve its performance, suggesting that graphical representations in particular are important to modeling human judgements. Taken together, these results suggest that relational graphical representations of visual information can help artificial vision systems make more human-like social judgments without incurring the computational cost of Bayesian models, and provide insights into the computations that humans employ while making visual social judgments.

Metacognitive understanding of visual motion cues to intentionality

Metacognitive understanding of visual motion cues to intentionality

Evidence for a consistent use of external cues by marine fish larvae for orientation

The larval stage is the main dispersive process of most marine teleost species. The degree to which larval behavior controls dispersal has been a subject of debate. Here, we apply a cross-species meta-analysis, focusing on the fundamental question of whether larval fish use external cues for directional movement (i.e., directed movement). Under the assumption that directed movement results in straighter paths (i.e., higher mean vector lengths) compared to undirected, we compare observed patterns to those expected under undirected pattern of Correlated Random Walk (CRW). We find that the bulk of larvae exhibit higher mean vector lengths than those expected under CRW, suggesting the use of external cues for directional movement. We discuss special cases which diverge from our assumptions. Our results highlight the potential contribution of orientation to larval dispersal outcomes. This finding can improve the accuracy of larval dispersal models, and promote a sustainable management of marine resources.

A New Prepositioning Technique of a Motion Simulator Platform Using Nonlinear Model Predictive Control and Recurrent Neural Network

The motion cueing algorithm (MCA) is the main algorithm in motion simulators in charge of generating vehicle motions within the platform’s constraints. The classical washout filter is one of the popular types of MCA, which is used in air and land vehicle motion simulators. The fixed home position of the simulator platform is always cogitated in the MCA to washout the motion simulator after generating each motion. Unfortunately, considering the fixed home position reduces the efficient consumption of the workspace in the linear directions. The linear motion of the motion simulator is due to the production of the high-pass frequency part of the motion scenarios. Prepositioning is used to tackle this assumption by varying the home position rather than the fixed position. The linear motion limitations of the motion simulator can virtually be enlarged using the prepositioning method. The efficient regeneration of the high-pass motion cues using a new propositioning technique is the main goal of this study to increase the motion realism of the simulator and remove any false motion cues due to the platform limitations. The proposed model utilised the recurrent neural network (RNN) to estimate the motion scenario along the prediction horizon. The nonlinear model predictive control (MPC) uses the estimated motion signals to extract the best optimal off-centre position of the motion simulator platform. The newly developed prepositioning technique is developed in the simulation environment of MATLAB to validate the proposed technique in terms of efficiency and applicability. The outcomes prove the capability of the proposed technique against the recently developed prepositioning technique using fuzzy logic and RNN.

Multi-object tracking with robust object regression and association

Tracking-by-regression is a new paradigm for online Multi-Object Tracking (MOT). It unifies detection and tracking into a single network by associating targets through regression, significantly reducing the complexity of data association. However, owing to noisy features from nearby occlusions and distractors, the regression is vulnerable and unaware of the inter-object occlusions and intra-class distractors. Thus the regressed bounding boxes can be wrongly suppressed or easily drift. Meanwhile, the commonly used bounding box-based post-processing is unable to remedy false negatives and false assignments caused by regression. To address these challenges, we present to leverage regression tubes as input for the regression-based tracker, which provides spatial–temporal information to enhance the tracking performance. Specially, we propose a novel tube re-localization strategy that obtains robust regressions and recovers missed targets. A tube-based NMS (T-NMS) strategy to manage the regressions at the tube level is also proposed, including a tube IoU (T-IoU) scheme for measuring positional relation and tube re-scoring (T-RS) to evaluate the quality of candidate tubes. Finally, a tube re-assignment strategy is further employed for robust cost measurement and to revise false assignments using motion cues. We evaluate our method on benchmarks, including MOT16, MOT17, and MOT20. The results show that our method can significantly improve the baseline, mitigate the challenges of the regression-based tracker, and achieve very competitive tracking performance.

Combination of visuo-tactile and visuo-vestibular correlations in illusory body ownership and self-motion sensations.

Previous studies have shown that illusory ownership over a mannequin's body can be induced through synchronous visuo-tactile stimulation as well as through synchronous visuo-vestibular stimulation. The current study aimed to elucidate how three-way combinations of correlated visual, tactile and vestibular signals contribute to the senses of body ownership and self-motion. Visuo-tactile temporal congruence was manipulated by touching the mannequin's body and the participant's unseen real body on the trunk with a small object either synchronously or asynchronously. Visuo-vestibular temporal congruence was manipulated by synchronous or asynchronous presentation of a visual motion cue (the background rotating around the mannequin in one direction) and galvanic stimulation of the vestibular nerve generating a rotation sensation (in the same direction). The illusory experiences were quantified using a questionnaire; threat-evoked skin-conductance responses (SCRs) provided complementary indirect physiological evidence for the illusion. Ratings on the illusion questionnaire statement showed significant main effects of synchronous visuo-vestibular and synchronous visuo-tactile stimulations, suggesting that both of these pairs of bimodal correlations contribute to the ownership illusion. Interestingly, visuo-tactile synchrony dominated because synchronous visuo-tactile stimulation combined with asynchronous visuo-vestibular stimulation elicited a body ownership illusion of similar strength as when both bimodal combinations were synchronous. Moreover, both visuo-tactile and visuo-vestibular synchrony were associated with enhanced self-motion perception; self-motion sensations were even triggered when visuo-tactile synchrony was combined with visuo-vestibular asynchrony, suggesting that ownership enhanced the relevance of visual information as a self-motion cue. Finally, the SCR results suggest that synchronous stimulation of either modality pair led to a stronger illusion compared to the asynchronous conditions. Collectively, the results suggest that visuo-tactile temporal correlations have a stronger influence on body ownership than visuo-vestibular correlations and that ownership boosts self-motion perception. We present a Bayesian causal inference model that can explain how visuo-vestibular and visuo-tactile information are combined in multisensory own-body perception.

Event-guided Multi-patch Network with Self-supervision for Non-uniform Motion Deblurring

Contemporary deep learning multi-scale deblurring models suffer from many issues: (I) They perform poorly on non-uniformly blurred images/videos; (II) Simply increasing the model depth with finer-scale levels cannot improve deblurring; (III) Individual RGB frames contain a limited motion information for deblurring; (IV) Previous models have a limited robustness to spatial transformations and noise. Below, we propose several mechanisms based on the multi-patch network to address the above issues: (I) We present a novel self-supervised event-guided deep hierarchical Multi-patch Network (MPN) to deal with blurry images and videos via fine-to-coarse hierarchical localized representations; (II) We propose a novel stacked pipeline, StackMPN, to improve the deblurring performance under the increased network depth; (III) We propose an event-guided architecture to exploit motion cues contained in videos to tackle complex blur in videos; (IV) We propose a novel self-supervised step to expose the model to random transformations (rotations, scale changes), and make it robust to Gaussian noises. Our MPN achieves the state of the art on the GoPro and VideoDeblur datasets with a 40 $$\times $$ faster runtime compared to current multi-scale methods. With 30 ms to process an image at 1280 $$\times $$ 720 resolution, it is the first real-time deep motion deblurring model for 720p images at 30 fps. For StackMPN, we obtain significant improvements over 1.2 dB on the GoPro dataset by increasing the network depth. Utilizing the event information and self-supervision further boost results to 33.83 dB.

Motion-region annotation for complex videos via label propagation across occluders

Motion cue is pivotal in moving object analysis, which is the root for motion segmentation and detection. These preprocessing tasks are building blocks for several applications such as recognition, matching and estimation. To devise a robust algorithm for motion analysis, it is imperative to have a comprehensive dataset to evaluate an algorithm’s performance. The main limitation in making these kind of datasets is the creation of ground-truth annotation of motion, as each moving object might span over multiple frames with changes in size, illumination and angle of view. Besides the optical changes, the object can undergo occlusion by static or moving occluders. The challenge increases when the video is captured by a moving camera. In this paper, we tackle the task of providing ground-truth annotation on motion regions in videos captured from a moving camera. With minimal manual annotation of an object mask, we are able to propagate the label mask in all the frames. Object label correction based on static and moving occluder is also performed by applying occluder mask tracking for a given depth ordering. A motion annotation dataset is also proposed to evaluate algorithm performance. The results show that our cascaded-naive approach provides successful results. All the resources of the annotation tool are publicly available at http://dixie.udg.edu/anntool/.

"Descriptive Risk-Averse Bayesian Decision-Making," a Model for Complex Biological Motion Perception in the Human Dorsal Pathway.

Biological motion perception is integral not only to survival but also to the social life of human beings. Identifying the underlying mechanisms and their associated neurobiological substrates has been a matter of investigation and debate for some time. Although, in general, it is believed that the integration of local motion and dynamic form cues in the brain empowers the visual system to perceive/recognize biological motion stimuli, some recent studies have indicated the importance of dynamic form cues in such a process. Inspired by the previous neurophysiologically plausible biological motion perception models, a new descriptive risk-averse Bayesian simulation model, capable of discerning a ball's direction from a set of complex biological motion soccer kick stimuli, is proposed. The model represents only the dorsal pathway as a motion information processing section of the visual system according to the two-stream theory. The stimuli used have been obtained from a previous psychophysical study on athletes in our lab. Furthermore, the acquired psychophysical data from that study have been used to re-enact human behavior using our simulation model. By adjusting the model parameters, the psychometric function of athlete subjects has been mimicked. A correlation analysis between human and simulation data shows a significant and robust correlation between angular thresholds and slopes of the psychometric functions of both groups. Although it is established that the visual system optimally integrates all available information in the decision-making process, the results conform to the speculations favoring motion cue importance over dynamic form by testing the limits in which biological motion perception only depends on motion information processing.

Effects of unlimited angular motion cue and cue discrepancy on simulator sickness

Simulator sickness is a crucial concern undermining several benefits of simulator training, such as a realistic environment, low costs, and safe practice of emergencies. This study investigated the effects of unbounded angular motions and visual-vestibular cue discrepancies on simulator sickness for flight simulator training. Human subject experiments with 36 participants demonstrated that simulator sickness, measured by questionnaires and physiological signals, was significantly decreased by offering both motion and visual cues rather than visual signals alone (p < 0.05). Specifically, nausea (without motion = 54.59, with motion = 31.27; p = 0.036) and disorientation scores (without motion = 81.20, with motion = 44.08; p = 0.028) significantly decreased when both motion and visual signals were present. Furthermore, the experimental results showed a significant correlation between simulator sickness and visual-vestibular cue mismatches, particularly for the angular velocity along the z-axis (r = 0.110, p = 0.04). The pitch angle discrepancy (r = 0.156, p = 0.004) between the visual and motion cues was significantly correlated with the sickness severity, unlike the roll angle disparity (r = −0.009, p = 0.871). The results from this study can be explored for flight training operations using motion simulators to minimize or eliminate simulator sickness.