Motion Cues Research Articles

Jointly modeling association and motion cues for robust infrared UAV tracking

Jointly modeling association and motion cues for robust infrared UAV tracking

Dynamic Spatial Focus for Efficient Compressed Video Action Recognition

Recent years have witnessed a growing interest in compressed video action recognition due to the rapid growth of online videos. It remarkably reduces the storage by replacing raw videos with sparsely sampled RGB frames and other compressed motion cues (motion vectors and residuals). However, existing compressed video action recognition methods face two main issues: First, the inefficiency caused by the usage of coarse-level information under full resolution, and second, the disturbing due to the noisy dynamics in motion vectors. To address the two issues, this paper proposes a dynamic spatial focus method for efficient compressed video action recognition (CoViFocus). Specifically, we first use a light-weighted two-stream architecture to localize the task-relevant patches for both the RGB frames and motion vectors. Then the selected patch pair will be processed by a high-capacity two-stream deep model for the final prediction. Such a patch selection strategy crops out the irrelevant motion noise in motion vectors, as well as reduces the spatial redundancy of the inputs, leading to the high efficiency of our method in the compressed domain. Moreover, we found that the motion vectors can help our method to address the possibly happened <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">static-issue</i> , which means that the focus patches get stuck at some regions related to static objects rather than target actions, which further improves our method. Extensive results on both the HMDB-51 and UCF-101 datasets demonstrate the effectiveness and efficiency of our method in compressed video action recognition tasks.

Kinematics modeling network for video-based human pose estimation

Estimating human poses from videos is critical in human–computer interaction. Joints cooperate rather than move independently during human movement. There are both spatial and temporal correlations between joints. Despite the positive results of previous approaches, most of them focus on modeling the spatial correlation between joints while only straightforwardly integrating features along the temporal dimension, which ignores the temporal correlation between joints. In this work, we propose a plug-and-play kinematics modeling module (KMM) to explicitly model temporal correlations between joints across different frames by calculating their temporal similarity. In this way, KMM can capture motion cues of the current joint relative to all joints in different time. Besides, we formulate video-based human pose estimation as a Markov Decision Process and design a novel kinematics modeling network (KIMNet) to simulate the Markov Chain, allowing KIMNet to locate joints recursively. Our approach achieves state-of-the-art results on two challenging benchmarks. In particular, KIMNet shows robustness to the occlusion. Code will be released at https://github.com/YHDang/KIMNet.

Efficient Video Transformers via Spatial-temporal Token Merging for Action Recognition

Transformer has exhibited promising performance in various video recognition tasks but brings a huge computational cost in modeling spatial-temporal cues. This work aims to boost the efficiency of existing video transformers for action recognition through eliminating redundancies in their tokens and efficiently learning motion cues of moving objects. We propose a lightweight and plug-and-play module, namely Spatial-temporal Token Merger (STTM), to merge the tokens belonging to the same object into a more compact representation. STTM first adaptively identifies crucial object clues underlying the video as meta tokens. Similarity scores between input tokens and meta tokens are hence computed and used to guide the fusion of similar tokens in both spatial and temporal domains, respectively. To compensate for motion cues lost in the merging procedure, we compute the linear aggregation of spatial-temporal positions of tokens as motion features. STTM hence outputs a compact set of tokens fusing both appearance and motion features of moving objects. This procedure substantially decreases the number of tokens that need to be processed by each Transformer block and boosts the efficiency. As a general module, STTM can be applied to different layers of various video Transformers. Extensive experiments on the action recognition datasets Kinectics-400 and SSv2 demonstrate its promising performance. For example, it reduces the computation complexity of ViT by 38% while maintaining a similar performance on Kinectics-400. It also brings 1.7% gains of top-1 accuracy on SSv2 under the same computational cost.

Morphology and spectral sensitivity of long visual fibers and lamina monopolar cells in the butterfly Papilio xuthus.

Extensive analysis of the flower-visiting behavior of a butterfly,Papilio xuthus, has indicated complex interaction between chromatic, achromatic, and motion cues. Their eyes are spectrally rich with six classes of photoreceptors, respectively sensitive in the ultraviolet, violet, blue, green, red, and broad-band wavelength regions. Here, we studied the anatomy and physiology of photoreceptors and second-order neurons ofP. xuthus, focusing on their spectral sensitivities and projection terminals to address where the early visual integration takes place. We thus found the ultraviolet, violet, and blue photoreceptors and all second-order neurons terminate in the distal region of the second optic ganglion, the medulla. We identified five types of second-order neurons based on the arborization in the first optic ganglion, the lamina, and the shape of the medulla terminals. Their spectral sensitivity is independent of the morphological types but reflects the combination of pre-synaptic photoreceptors. The results indicate that the distal medulla is the most plausible region for early visual integration.

Effect of Normal Force Intensity on Tactile Motion Speed Perception Based on Spatiotemporal Cue.

While the relative motion between the skin and objects in contact with it is essential to everyday tactile experiences, our understanding of how tactile motion is perceived via human tactile function is limited. Previous studies have explored the effect of normal force on speed perception under conditions where multiple motion cues on the skin (spatiotemporal cue, tangential skin deformation cue, and slip-induced vibration cue) were integrated. However, the effect of the normal force on speed perception in terms of each motion cue remains unclear since the multiple motion cues have not been adequately separated in the previously reported experiments. In this study, we aim to elucidate the effect of normal force in situations where the speed perception of tactile motion is based solely on a spatiotemporal cue. We developed a pin-array display which allowed us to vary the intensity of the normal force without causing tangential forces or slip-induced vibrations. Using the display, we conducted two psychophysical experiments. In Experiment 1, we found that the speed of the object was perceived to be 1.12-1.14 times faster when the intensity of the normal force was doubled. In Experiment 2, we did not observe significant differences in the discriminability of tactile speed caused by differences in normal force intensity. Our experimental results are of scientific significance and offer insights for engineering applications when using haptic displays that can only provide spatiotemporal cues represented by normal forces.

An Adaptive Spatio-temporal Global Sampling for Presentation Attack Detection

Without developing dedicated countermeasures, facial biometric systems can be spoofed with printed photos, replay attacks, silicone masks, or even a 3D mask of a targeted person. Thus, the threat of presentation attacks needs to be addressed to strengthen the security of the biometric systems. Since a 2D convolutional neural network (CNN) captures static features from video frames, the camera motion might hinders the performance of modern CNNs for video-based presentation attack detection (PAD). Inspired by the egomotion theory, we introduce an adaptive spatiotemporal global sampling (ASGS) technique to compensate the camera motion and use the resulting estimation to encode the appearance and dynamics of the video sequences into a single RGB image. This is achieved by adaptively splitting the video into small segments and capturing their global motion within each segment. The proposed global motion is estimated based on four key steps: dense sampling, FREAK feature extraction and matching, similarity transformation, and aggregation function. This allows using deep models pre-trained on images for video-based PAD detection. Moreover, the interpretation of ASGS reveals that the most important parts for supporting the decision on PAD are consistent with motion cues associated with the artifacts, i.e., hand movement, material reflection, and expression changes. Extensive experiments on four standard face PAD databases demonstrate its effectiveness and encourage further study in this domain.

Dynamic-guided Spatiotemporal Attention for Echocardiography Video Segmentation.

Left ventricle (LV) endocardium segmentation in echocardiography video has received much attention as an important step in quantifying LV ejection fraction. Most existing methods are dedicated to exploiting temporal information on top of 2D convolutional networks. In addition to single appearance semantic learning, some research attempted to introduce motion cues through the optical flow estimation (OFE) task to enhance temporal consistency modeling. However, OFE in these methods is tightly coupled to LV endocardium segmentation, resulting in noisy inter-frame flow prediction, and post-optimization based on these flows accumulates errors. To address these drawbacks, we propose dynamic-guided spatiotemporal attention (DSA) for semi-supervised echocardiography video segmentation. We first fine-tune the off-the-shelf OFE network RAFT on echocardiography data to provide dynamic information. Taking inter-frame flows as additional input, we use a dual-encoder structure to extract motion and appearance features separately. Based on the connection between dynamic continuity and semantic consistency, we propose a bilateral feature calibration module to enhance both features. For temporal consistency modeling, the DSA is proposed to aggregate neighboring frame context using deformable attention that is realized by offsets grid attention. Dynamic information is introduced into DSA through a bilateral offset estimation module to effectively combine with appearance semantics and predict attention offsets, thereby guiding semantic-based spatiotemporal attention. We evaluated our method on two popular echocardiography datasets, CAMUS and EchoNet-Dynamic, and achieved state-of-the-art.

Consideration of the problem of motion cueing along angular degrees of freedom on flight simulators

The object of research is motion cueing along angular degrees of freedom on flight simulators of non-maneuvering aircraft. One of the most problematic places is lack of statement and effective solution of the problem to ensure high-quality motion cues along angular degrees of freedom on flight simulators, which would correspond to motion cues along angular degrees of freedom in real flight with the same control actions. In the course of the research, on the basis of the peculiarities of human movement perception, a set of characteristic attributes of perception of motion cues is determined: character, direction, duration, intensity and time of motion perception (according to Gibson’s perception theory). Based on the system approach principles, the mathematical formulation of the solution to the problem of motion cueing along angular degrees of freedom on flight simulators of non-maneuvering aircraft is used. Such approach made it possible, taking into account the existing constructive resource of flight simulator motion system, to bring as close as possible motion cueing along angular degrees of freedom on flight simulators of non-maneuvering aircraft to motion cues along angular degrees of freedom in real flight with the same control actions. Due to this the character and direction of motion cues fully correspond to the real motion cues, the difference between the perception time of motion cues on airplane and simulator is minimal and meets the current requirements. The duration and intensity of the motion cue perception on simulator are proportional duration and intensity of motion cue perception on airplane. Such approach significantly improves the quality of training and retraining of pilots on flight simulators. Implementation of the developed problem formulation on aircraft simulators, in particular on An-74TK-200, showed its high efficiency. In the future, the proposed approach can be used on flight simulators of aircraft developed in Ukraine and modernization of operated flight simulators.

Unsupervised video object segmentation for enhanced SLAM-based localization in dynamic construction environments

Accurate robot localization in dynamic construction environments is vital for successful robotics implementation on jobsites. Dynamic Simultaneous Localization and Mapping (SLAM) methods show promise in addressing challenges by identifying and removing dynamic objects. However, existing studies struggle to accurately segment these objects, primarily due to handling unseen and temporarily static objects. This study presents a novel approach for robustly segmenting dynamic objects in construction environments. The approach captures potential objects using objectness masks and refines them by fusing masks with motion cues. The segmented information is propagated bi-directionally across frames. The method comprises four stages: objectness masks generation, motion saliency estimation, fusion of objectness masks and motion saliency, and bi-directional propagation of fused masks. Experimental results demonstrate the method's effectiveness in accurately segmenting dynamic objects and improving localization accuracy when integrated with SLAM structure, highlighting its potential for real-world applications in complex construction environments.

Impact of motion cues, color, and luminance on depth perception in optical see-through AR displays

Impact of motion cues, color, and luminance on depth perception in optical see-through AR displays

Model Predictive Control based Motion Cueing Algorithm for Driving Simulator

Model Predictive Control based Motion Cueing Algorithm for Driving Simulator

Visual Cue Effects on a Classification Accuracy Estimation Task in Immersive Scatterplots.

Immersive visualization in virtual reality (VR) allows us to exploit visual cues for perception in 3D space, yet few existing studies have measured the effects of visual cues. Across a desktop monitor and a head-mounted display (HMD), we assessed scatterplot designs which vary their use of visual cues-motion, shading, perspective (graphical projection), and dimensionality-on two sets of data. We conducted a user study with a summary task in which 32 participants estimated the classification accuracy of an artificial neural network from the scatterplots. With Bayesian multilevel modeling, we capture the intricate visual effects and find that no cue alone explains all the variance in estimation error. Visual motion cues generally reduce participants' estimation error; besides this motion, using other cues may increase participants' estimation error. Using an HMD, adding visual motion cues, providing a third data dimension, or showing a more complicated dataset leads to longer response times. We speculate that most visual cues may not strongly affect perception in immersive analytics unless they change people's mental model about data. In summary, by studying participants as they interpret the output from a complicated machine learning model, we advance our understanding of how to use the visual cues in immersive analytics.

Investigation of vibration's effect on driver in optimal motion cueing algorithm.

The increased sensation error between the surroundings and the driver is a major problem in driving simulators, resulting in unrealistic motion cues. Intelligent control schemes have to be developed to provide realistic motion cues to the driver. The driver's body model incorporates the effects of vibrations on the driver's health, comfort, perception, and motion sickness, and most of the current research on motion cueing has not considered these factors. This article proposes a novel optimal motion cueing algorithm that utilizes the driver's body model in conjunction with the driver's perception model to minimize the sensation error. Moreover, this article employs H∞ control in place of the linear quadratic regulator to optimize the quadratic cost function of sensation error. As compared to state of the art, we achieve decreased sensation error in terms of small root-mean-square difference (70%, 61%, and 84% decrease in case of longitudinal acceleration, lateral acceleration, and yaw velocity, respectively) and improved coefficient of cross-correlation (3% and 1% increase in case of longitudinal and lateral acceleration, respectively).

Interaction Design and Research for User Interfaces of Central Bank Digital Currency: An Empirical Study

There is already a consensus among central banks to accelerate the development of Central Bank Digital Currencies (CBDC). To facilitate a smooth money transfer process, the interactive experience of the CBDC user interface design is very important. However, much of the current research has focused on the technical feasibility of the CBDCs, and no scientific research has yet attempted to design an effective CBDC user interface. In this study, a series of CBDC user interfaces were designed to help study the interaction process of CBDCs in a simulated payment transaction context with a focus on the user interface and interactive experience of the CBDCs. The experiment in this study was a 2 (graphical metaphor) × 3 (gesture type) × 2 (visual feedback) mixed factorial design in which a total of 12 CBDC prototypes were created and evaluated based on users’ task performance and a series of subjective evaluations. The results of our empirical study revealed that graphical metaphor user interface design can improve the interface usability and subjective evaluation of CBDCs, and may also trigger the user’s sense of experience with traditional banknotes. Specifically, the use of animated cues for currency movement on the CBDC user interface may help evoke the user’s experience of using cash. In addition, we found that the wave type gesture may cause users to spend more time completing the CBDC operational tasks. Moreover, the animated cue of money moving may have faster performance and a higher degree of rationality. The above-mentioned results may provide an important design reference for CBDC developers to create easy-to-use CBDC services.

Neck stabilization through sensory integration of vestibular and visual motion cues.

To counteract gravity, trunk motion, and other perturbations, the human head-neck system requires continuous muscular stabilization. In this study, we combine a musculoskeletal neck model with models of sensory integration (SI) to unravel the role of vestibular, visual, and muscle sensory cues in head-neck stabilization and relate SI conflicts and postural instability to motion sickness. A 3D multisegment neck model with 258 Hill-type muscle elements was extended with postural stabilization using SI of vestibular (semicircular and otolith) and visual (rotation rate, verticality, and yaw) cues using the multisensory observer model (MSOM) and the subjective vertical conflict model (SVC). Dynamic head-neck stabilization was studied using empirical datasets, including 6D trunk perturbations and a 4 m/s2 slalom drive inducing motion sickness. Recorded head translation and rotation are well matched when using all feedback loops with MSOM or SVC or assuming perfect perception. A basic version of the model, including muscle, but omitting vestibular and visual perception, shows that muscular feedback can stabilize the neck in all conditions. However, this model predicts excessive head rotations in conditions with trunk rotation and in the slalom. Adding feedback of head rotational velocity sensed by the semicircular canals effectively reduces head rotations at mid-frequencies. Realistic head rotations at low frequencies are obtained by adding vestibular and visual feedback of head rotation based on the MSOM or SVC model or assuming perfect perception. The MSOM with full vision well captures all conditions, whereas the MSOM excluding vision well captures all conditions without vision. The SVC provides two estimates of verticality, with a vestibular estimate SVCvest, which is highly effective in controlling head verticality, and an integrated vestibular/visual estimate SVCint which can complement SVCvest in conditions with vision. As expected, in the sickening drive, SI models imprecisely estimate verticality, resulting in sensory conflict and postural instability. The results support the validity of SI models in postural stabilization, where both MSOM and SVC provide credible results. The results in the sickening drive show imprecise sensory integration to enlarge head motion. This uniquely links the sensory conflict theory and the postural instability theory in motion sickness causation.

When "looking at nothing" imparts something: Retrospective gaze cues flexibly direct prioritization in visual working memory.

Previous research has shown that retrospective gaze cues direct attention to internally maintained representations in visual working memory (vWM). Here, we aimed to differentiate the dual nature of gaze and accordingly proposed two hypotheses regarding the gaze-induced prioritization in vWM. The directional cueing hypothesis claims a constant attentional shifting to the gazed-at direction. By contrast, the referential cueing hypothesis proposes that gaze cues selectively orient attention toward their referents. To test these hypotheses, we employed an adapted change-detection task wherein gaze cues were presented during the retention interval. Critically, the cue character was positioned between two barriers, which could be either opaque (the blocked condition) or transparent (the unblocked condition). Polygons previously presented at the gazed-at (vs. gazed-away) location were better memorized, but not when the visual perspective of the character was obstructed (i.e., the blocked condition, Experiment 1). Subsequent experiments demonstrated that physical motion cues (Experiment 2) and inverted face cues (Experiment 3), which disrupted the extraction of referential signals, were immune to barrier settings. In Experiment 4, we generalize this selective cueing effect to faces with fearful expressions. These consistent findings support the referential cueing hypothesis and emphasize the distinctiveness of social attention. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Inducers of motion sickness in vehicles: A systematic review of experimental evidence and meta-analysis

With the development of intelligent cabins and the popularity of smart devices, motion sickness has become an increasing concern. A number of theories have been proposed to explain the cause of motion sickness in vehicles. However, because of the diversity of the dynamical characteristics, road conditions, and in-vehicle human–machine interaction designs, the influence of the inducers on motion sickness in vehicles has not yet been fully quantified. Thus, in this paper, we aim to review and summarize the influential factors of motion sickness in road vehicles through a systematic review. In total, we identified 57 studies related to influential factors of motion sickness in vehicles, of which, 27 were further included for meta-analysis. In total, we identified eight categories of motion sickness inducers in vehicles, including the type of eye view, non-driving-related task availability, existence of artificial motion cues, head dynamic movement, vehicle dynamics, internal layout of vehicle, individual differences, and others. Most inducers had consistent effects on inducing motion sickness; however, inconsistent results have also been observed in vehicle dynamics, head dynamic movement, eye view, individual difference, and artificial motion cues (i.e., cues regarding the motion states of the ego-vehicle provided through in-vehicle auditory, tactile, or visual interfaces). Additional meta-analysis was conducted for motion cues. It was found that only natural present motion cues (i.e., cues regarding relative movement between the ego-vehicle and the environment through the windshield or side windows) and non-visual artificial anticipatory motion cues (i.e., auditory or tactile artificial motion cues regarding the future motion of ego-vehicle) were effective in alleviating motion sickness. Future research directions have been pointed out. Our study can provide insights into the optimization of vehicle design to mitigate motion sickness among occupants.

FishTrack: Multi-object tracking method for fish using spatiotemporal information fusion

Tracking the fish is a key step in analyzing fish behavior, evaluating their health levels, and warning of abnormal water quality, so it is of significant importance for intelligent monitoring in fish farming. However, multi-object tracking for fish is a very challenging task due to foam occlusion in the factory tank, high individual similarity, and fast-paced motion of fish. To address the above issues, an online multi-fish tracking model called FishTrack is proposed with 3 branches of target detection, track prediction, and re-identification, which establishes the motion model and appearance model for fish simultaneously, thus achieving the online multi-fish tracking. It is a complete Encoder-Decoder structure and the Pyramid Vision Transformer (PVT) is adopted as the backbone network to extract multi-level features. Then an Encoder is specially designed to encode the historical information of the positions of fish, and automatically update its spatiotemporal information in an autoregressive fashion, to perform the fusion of the spatiotemporal information of fish targets and avoid manual selection of spatiotemporal features. Finally, a parallel dual-Decoder is used to decode the motion and appearance features of fish to reduce the interference of the two optimization directions during the joint model training. The motion cues are first used in linear assignment to reduce the interference of occlusion and deformation issues, and when the motion model fails to track, the appearance cues can recover the identity to deal with long occlusion. The experiments on the established multiple fish tracking dataset showed that the higher order tracking accuracy (HOTA) reached 71.4%, the multi-object tracking accuracy (MOTA) reached 94.8%, the most tracked ratio (MT) reached 93.3%, and the identification F1 score (IDF1) reached 82.5%. The results show that FishTrack can solve the problem of tracking accuracy decrease caused by foam occlusion and fish deformation, and save the inference time by sharing features, increasing the robustness of online multi-fish tracking in factory farming.

Novel model predictive control-based motion cueing algorithm for compensating centrifugal acceleration in KUKA robocoaster-based driving simulators.

The washout motion cueing algorithm (MCA) is a critical element in driving simulators, designed to faithfully reproduce precise motion cues while minimizing false cues during simulation processes, particularly deceptive translational and rotational cues. To enhance motion sensation accuracy and optimize the use of available workspace, model predictive control (MPC) has been employed to develop innovative motion cueing algorithms. While most MCAs have been tailored for the Steward motion platform, there has been a recent adoption of the motion platform based on KUKA Robocoaster as an economical option for driving simulators. However, leveraging the full potential of the KUKA Robocoaster requires trajectory conversion of the motion base. Thus, this research proposes a novel MCA specifically designed for the KUKA Robocoaster-based motion platform, utilizing large planar circular motion to simulate lateral movement for drivers. Nonetheless, circular motion introduces disruptive centrifugal forces, which can be mitigated through proper pitch-tilted angles. The novel MPC generates simulated motion that accurately follows the lateral specific force target and effectively maintains the roll angular velocity below its threshold value. Additionally, it compensates for disturbing centrifugal acceleration by implementing pitch rotational motion, ensuring the pitch angular velocity remains below its threshold. Simulation tasks conducted on the motion platform, focusing solely on lateral acceleration, demonstrate the successful elimination of false motion cues in both the roll/sway and pitch/surge channels. The proposed innovative MPC solution offers an original approach to motion cueing algorithms in KUKA Robocoaster-based driving simulators. It enables the exploitation of the KUKA Robocoaster platform's capabilities while delivering accurate and immersive motion cues to drivers during simulation experiences.