Motion-related Information Research Articles

MRDFlow: Unsupervised Optical Flow Estimation Network With Multi-Scale Recurrent Decoder

Optical flow estimation is a fundamental task in computer vision and image processing. Due to the difficulty in obtaining the ground truth of flow field, unsupervised learning approaches attract more and more research interests in recent years. However, despite of their good generalization capability, unsupervised optical flow methods suffer in the scenarios with large displacement, small objects, and occlusions. In this work, we propose a novel optical flow network based on decoder with multi-scale kernels. Different from previous U-Net like or pyramidal methods, we design our network based on RAFT architecture that with a 4D correlation layer and recurrent decoder. More importantly, we incorporate three novel ideas with regard to the input, information processing and output of the update units improve the performance. Firstly, we utilize various motion-related information as input to the update units. Secondly, we propose a module of multi-scale update unit. Thirdly, for the final flow up-sampling procedure, we propose an image-guided up-sampling loss to guide the learning of up-sampling masks. Our model is trained by the occlusion-aware photometric loss, edge-aware smoothness loss, self-supervised loss, and image-guided up-sampling loss. Experimental results demonstrate that our model achieves the state-of-the-art performance on both Sintel and KITTI and outperforms other unsupervised optical flow methods remarkably.

Zero-Shot Learning for IMU-Based Activity Recognition Using Video Embeddings

The Activity Recognition Chain generally precludes the challenging scenario of recognizing new activities that were unseen during training, despite this scenario being a practical and common one as users perform diverse activities at test time. A few prior works have adopted zero-shot learning methods for IMU-based activity recognition, which work by relating seen and unseen classes through an auxiliary semantic space. However, these methods usually rely heavily on a hand-crafted attribute space which is costly to define, or a learnt semantic space based on word embedding, which lacks motion-related information crucial for distinguishing IMU features. Instead, we propose a strategy to exploit videos of human activities to construct an informative semantic space. With our approach, knowledge from state-of-the-art video action recognition models is encoded into video embeddings to relate seen and unseen activity classes. Experiments on three public datasets find that our approach outperforms other learnt semantic spaces, with an additional desirable feature of scalability, as recognition performance is seen to scale with the amount of data used. More generally, our results indicate that exploiting information from the video domain for IMU-based tasks is a promising direction, with tangible returns in a zero-shot learning scenario.

Constructing assembly design model capable of capturing and sharing semantic dynamic motion information in heterogeneous CAD systems

Over the last decades, collaborative assembly design has received much attention, which has provided fruitful results in terms of data models, knowledge-based, and top-down geometric modeling approaches, to name a few. Particularly, researchers have focused their efforts towards assembly relationships within product models, but have limited the scope to static assembly geometric description as currently stated in computer-aided design (CAD) systems. A disruptive vision is assumed in this research work, arguing that the assembly design itself presents a dynamic nature via, especially, dynamic motion. Such a paradigm shift requires an appropriate logical and semantic foundation to augment current CAD systems and data models’ capabilities with the capture of dynamic phenomena such as assembly motions and kinematic behaviors. Traditional CAD systems nowadays can handle the kinematics-related motion-related information; however, the interoperation of this motion-related information from one CAD system to the other is still a limitation of the interoperability standards. To tackle this research challenge, this paper aims at defining an original framework built upon a formal ontology to capture and share dynamic assembly motion among heterogeneous CAD systems. The formal ontology uses spatiotemporal mereotopology, which serves as a theoretical backbone to qualitatively describe the dynamic behavior of a product. This theory addresses the qualitative description of the temporal dimension over the spatial dimension of 3D objects. The proposed ontology is then enhanced to maintain semantic continuity of the spatial and temporal aspects embedded in assembly design ontology models. Built with a Web Ontology Language (OWL) data structure to interact with a CAD system and a motion visualization application, the ontology ensures the dynamic motion information capturing and sharing. SWRL (Semantic Web Rule Language) is utilized to show the semantic reasoning among the moving and non-moving parts of the product assembly. As a result, a semantic articulation ensuring flexibility and interoperability as a competitive industrial edge is demonstrated with proof of concept to be effective in a collaborative assembly design environment.

Using Machine Learning and Smartphone and Smartwatch Data to Detect Emotional States and Transitions: Exploratory Study.

BackgroundEmotional state in everyday life is an essential indicator of health and well-being. However, daily assessment of emotional states largely depends on active self-reports, which are often inconvenient and prone to incomplete information. Automated detection of emotional states and transitions on a daily basis could be an effective solution to this problem. However, the relationship between emotional transitions and everyday context remains to be unexplored.ObjectiveThis study aims to explore the relationship between contextual information and emotional transitions and states to evaluate the feasibility of detecting emotional transitions and states from daily contextual information using machine learning (ML) techniques.MethodsThis study was conducted on the data of 18 individuals from a publicly available data set called ExtraSensory. Contextual and sensor data were collected using smartphone and smartwatch sensors in a free-living condition, where the number of days for each person varied from 3 to 9. Sensors included an accelerometer, a gyroscope, a compass, location services, a microphone, a phone state indicator, light, temperature, and a barometer. The users self-reported approximately 49 discrete emotions at different intervals via a smartphone app throughout the data collection period. We mapped the 49 reported discrete emotions to the 3 dimensions of the pleasure, arousal, and dominance model and considered 6 emotional states: discordant, pleased, dissuaded, aroused, submissive, and dominant. We built general and personalized models for detecting emotional transitions and states every 5 min. The transition detection problem is a binary classification problem that detects whether a person’s emotional state has changed over time, whereas state detection is a multiclass classification problem. In both cases, a wide range of supervised ML algorithms were leveraged, in addition to data preprocessing, feature selection, and data imbalance handling techniques. Finally, an assessment was conducted to shed light on the association between everyday context and emotional states.ResultsThis study obtained promising results for emotional state and transition detection. The best area under the receiver operating characteristic (AUROC) curve for emotional state detection reached 60.55% in the general models and an average of 96.33% across personalized models. Despite the highly imbalanced data, the best AUROC curve for emotional transition detection reached 90.5% in the general models and an average of 88.73% across personalized models. In general, feature analyses show that spatiotemporal context, phone state, and motion-related information are the most informative factors for emotional state and transition detection. Our assessment showed that lifestyle has an impact on the predictability of emotion.ConclusionsOur results demonstrate a strong association of daily context with emotional states and transitions as well as the feasibility of detecting emotional states and transitions using data from smartphone and smartwatch sensors.

Stochastic Modeling Based Nonlinear Bayesian Filtering for Photoplethysmography Denoising in Wearable Devices

Photoplethysmography (PPG) has shown its great potential for noninvasive health monitoring, but its application in wearable devices is largely impeded due to its extreme vulnerability to motion artifacts. In this article, we proposed a new stochastic modeling based nonlinear Bayesian filtering framework for the recovery of corrupted PPG waveform under strenuous physical exercise in wearable health-monitoring devices. A deep recurrent neural network was first recruited for accurate cardiac-period segmentation of corrupted PPG signals. Then, a stochastic model was applied to extract waveform details from clean PPG pulses, and was further derived into a system-state space. Following this was an extended Kalman filter using the state-space structured by modeling. The covariance of measurement noise was estimated by motion-related information to adjust it into the real physical environment adaptively. Comparison results with state-of-the-art methods on a wearable-device-based 48-subject data set showed the outstanding performance of the proposed denoising framework, with period-segmentation sensitivity and precision higher than 99.1%, instantaneous heart rate (HR) error lower than 2 beats/min, average HR error down to 1.14 beats/min, and recovery accuracy of waveform details significantly improved (p < 0.05). This framework is the first PPG denoising strategy that introduces waveform-modeling methods to ensure detail recovery, and a great example of algorithm fusion between stochastic signal processing and emerging deep learning methods for time-sequential biomedical signal processing.

A review of Convolutional-Neural-Network-based action recognition

Video action recognition is widely applied in video indexing, intelligent surveillance, multimedia understanding, and other fields. Recently, it was greatly improved by incorporating the learning of deep information using Convolutional Neural Network (CNN). This motivated us to review the notable CNN-based action recognition works. Because CNN is primarily designed to extract 2D spatial features from still image and videos are naturally viewed as 3D spatiotemporal signals, the core issue of extending the CNN from image to video is temporal information exploitation. We divide the solutions for exploiting temporal information exploration into three strategies: 1) 3D CNN; 2) taking the motion-related information as the CNN input; and 3) fusion. In this paper, we present a comprehensive review of the CNN-based action recognition methods according to these strategies. We also discuss the action recognition performance on recent large-scale benchmarks and the limitations and future research directions of CNN-based action recognition. This paper offers an objective and clear review of CNN-based action recognition and provides a guide for future research.

Motion of glossy objects does not promote separation of lighting and surface colour.

The surface properties of an object, such as texture, glossiness or colour, provide important cues to its identity. However, the actual visual stimulus received by the eye is determined by both the properties of the object and the illumination. We tested whether operational colour constancy for glossy objects (the ability to distinguish changes in spectral reflectance of the object, from changes in the spectrum of the illumination) was affected by rotational motion of either the object or the light source. The different chromatic and geometric properties of the specular and diffuse reflections provide the basis for this discrimination, and we systematically varied specularity to control the available information. Observers viewed animations of isolated objects undergoing either lighting or surface-based spectral transformations accompanied by motion. By varying the axis of rotation, and surface patterning or geometry, we manipulated: (i) motion-related information about the scene, (ii) relative motion between the surface patterning and the specular reflection of the lighting, and (iii) image disruption caused by this motion. Despite large individual differences in performance with static stimuli, motion manipulations neither improved nor degraded performance. As motion significantly disrupts frame-by-frame low-level image statistics, we infer that operational constancy depends on a high-level scene interpretation, which is maintained in all conditions.

Analysis of construction trade worker body motions using a wearable and wireless motion sensor network

Abstract Biomechanical analysis of construction workers has been considerably improved with the development of wearable sensors. Information delivered by these systems is playing an important role in the evaluation of postures as well as in the reduction of work-related musculoskeletal disorders (WRMSDs). In this article, we present a novel system and data processing framework to deliver intuitive and understandable motion-related information about workers. The system uniquely integrates Inertial Measurement Unit (IMU) devices in a wireless body area network, and the data processing uses a robust state machine-based approach that assesses inadequate working postures based on standard positions defined by the International Organization for Standardization (ISO). The system and data processing framework are collectively validated through experiments carried out with college trainees conducting typical bricklaying tasks. The results illustrate the robustness of the system under demanding circumstances, and suggest its applicability in actual working environments outside the college.

The effect of alexithymia on early visual processing of emotional body postures

Body postures convey emotion and motion-related information useful in social interactions. Early visual encoding of body postures, reflected by the N190 component, is modulated both by motion (i.e., postures implying motion elicit greater N190 amplitudes than static postures) and by emotion-related content (i.e., fearful postures elicit the largest N190 amplitude). At a later stage, there is a fear-related increase in attention, reflected by an early posterior negativity (EPN) (Borhani et al., 2015). Here, we tested whether difficulties in emotional processing (i.e., alexithymia) affect early and late visual processing of body postures. Low alexithymic participants showed emotional modulation of the N190, with fearful postures specifically enhancing N190 amplitude. In contrast, high alexithymic participants showed no emotional modulation of the N190. Both groups showed preserved encoding of the motion content. At a later stage, a fear-related modulation of the EPN was found for both groups, suggesting that selective attention to salient stimuli is the same in both low and high alexithymia.

The visual influence on path reproduction in darkness is stronger during childhood

Studying how we retrace our way in darkness provides a means of understanding the internal representations we use during navigation. In adults, path integration in darkness is influenced by previously presented visual information that conflicted with actual motion (Tcheang et al, PNAS, 2011, 108(3): 1152-7). Here we used immersive virtual reality to isolate visual and motor cues during encoding of a path in children and adults, and examined whether the cues' relative contributions to the stored path representation reflect optimal integration. Eighteen adults and fifteen 10-11-year-old children were guided along a two-leg path in darkness (motor-only), in a virtual room (visual-motor), or they watched a pre-recorded walk in the virtual room while standing still (visual-only). Participants then reproduced the path in darkness. By fitting bivariate normal distributions we obtained a measure of the dispersion of the end-points (variable error) and of their distances from the correct end point (constant error). While adults performed similarly across all three conditions, 10-11-year-old children reduced their variable error when encoding in the bimodal condition, indicating integration of cues. Both adults and children showed greater constant error in the visual-only condition than the motor-only (although significantly greater in adults than children), while the constant error for the visual-motor was intermediate between these two. A significant inverse correlation between visual-motor constant error and number of trials indicated that with only 10 repetitions children learnt to better combine the two cues. These results indicate that 10-11 year-old children can learn to optimally combine visual and motor information to encode a path. Adults' inability to do this could reflect their complete reliance on internal motion-related information because of its greater perceived reliability and relevance to the task. This suggests that the representations we use to navigate in darkness change significantly during the lifespan.

Human area MT+ shows load-dependent activation during working memory maintenance with continuously morphing stimulation.

BackgroundInitially, human area MT+ was considered a visual area solely processing motion information but further research has shown that it is also involved in various different cognitive operations, such as working memory tasks requiring motion-related information to be maintained or cognitive tasks with implied or expected motion.In the present fMRI study in humans, we focused on MT+ modulation during working memory maintenance using a dynamic shape-tracking working memory task with no motion-related working memory content. Working memory load was systematically varied using complex and simple stimulus material and parametrically increasing retention periods. Activation patterns for the difference between retention of complex and simple memorized stimuli were examined in order to preclude that the reported effects are caused by differences in retrieval.ResultsConjunction analysis over all delay durations for the maintenance of complex versus simple stimuli demonstrated a wide-spread activation pattern. Percent signal change (PSC) in area MT+ revealed a pattern with higher values for the maintenance of complex shapes compared to the retention of a simple circle and with higher values for increasing delay durations.ConclusionsThe present data extend previous knowledge by demonstrating that visual area MT+ presents a brain activity pattern usually found in brain regions that are actively involved in working memory maintenance.

Adaptively imperceptible video watermarking based on the local motion entropy

This paper proposes an adaptively imperceptible video watermarking algorithm using the entropy model for local motion characterization. The algorithm firstly combines Human Visual System (HVS) with the block- matching techniques to get the motion-related information. Then it utilizes the entropy model to statistically analyze above motion-related information to obtain the motion entropy of frame. Successively, this algorithm divides every frame into local regions, and then local motion entropy can be obtained according to the motion-related information in local region. By combining the local motion entropy with the motion entropy of frame, the motion characteristics visual masking is adaptively calculated. Based on the motion characteristics visual masking and the contents of video frames, the maximum strength of watermarking is calculated in every block. Experiments indicate that using entropy to local motion characteristics can significantly improve the watermarking imperceptibility, effectively resist common attacks for video watermarking and consequently achieve higher robustness.

Gaze direction affects linear self-motion heading discrimination in humans

We investigated the effect of eye-in-head and head-on-trunk direction on heading discrimination. Participants were passively translated in darkness along linear trajectories in the horizontal plane deviating 2° or 5° to the right or left of straight-ahead as defined by the subject's trunk. Participants had to report whether the experienced translation was to the right or left of the trunk straight-ahead. In a first set of experiments, the head was centered on the trunk and fixation lights directed the eyes 16° either left or right. Although eye position was not correlated with the direction of translation, rightward reports were more frequent when looking right than when looking left, a shift of the point of subjective equivalence in the direction opposite to eye direction (two of the 38 participants showed the opposite effect). In a second experiment, subjects had to judge the same trunk-referenced trajectories with head-on-trunk deviated 16° left. Comparison with the performance in the head-centered paradigms showed an effect of the head in the same direction as the effect of eye eccentricity. These results can be qualitatively described by biases reflecting statistical regularities present in human behaviors such as the alignment of gaze and path. Given the known effects of gaze on auditory localization and perception of straight-ahead, we also expect contributions from a general influence of gaze on the head-to-trunk reference frame transformations needed to bring motion-related information from the head-centered otoliths into a trunk-referenced representation.

Evidence of a direct influence between the thalamus and hMT+ independent of V1 in the human brain as measured by fMRI

In the present study we employed Conditional Granger Causality (CGC) and Coherence analysis to investigate whether visual motion-related information reaches the human middle temporal complex (hMT+) directly from the Lateral Geniculate Nucleus (LGN) of the thalamus, by-passing the primary visual cortex (V1). Ten healthy human volunteers underwent brain scan examinations by functional magnetic resonance imaging (fMRI) during two optic flow experiments. In addition to the classical LGN-V1-hMT+ pathway, our results showed a significant direct influence of the blood oxygenation level dependent (BOLD) signal recorded in LGN over that in hMT+, not mediated by V1 activity, which strongly supports the existence of a bilateral pathway that connects LGN directly to hMT+ and serves visual motion processing. Furthermore, we evaluated the relative latencies among areas functionally connected in the processing of visual motion. Using LGN as a reference region, hMT+ exhibited a statistically significant earlier peak of activation as compared to V1. In conclusion, our findings suggest the co-existence of an alternative route that directly links LGN to hMT+, bypassing V1. This direct pathway may play a significant functional role for the faster detection of motion and may contribute to explain persistence of unconscious motion detection in individuals with severe destruction of primary visual cortex (blindsight).

Visual influence on path integration in darkness indicates a multimodal representation of large-scale space

Our ability to return to the start of a route recently performed in darkness is thought to reflect path integration of motion-related information. Here we provide evidence that motion-related interoceptive representations (proprioceptive, vestibular, and motor efference copy) combine with visual representations to form a single multimodal representation guiding navigation. We used immersive virtual reality to decouple visual input from motion-related interoception by manipulating the rotation or translation gain of the visual projection. First, participants walked an outbound path with both visual and interoceptive input, and returned to the start in darkness, demonstrating the influences of both visual and interoceptive information in a virtual reality environment. Next, participants adapted to visual rotation gains in the virtual environment, and then performed the path integration task entirely in darkness. Our findings were accurately predicted by a quantitative model in which visual and interoceptive inputs combine into a single multimodal representation guiding navigation, and are incompatible with a model of separate visual and interoceptive influences on action (in which path integration in darkness must rely solely on interoceptive representations). Overall, our findings suggest that a combined multimodal representation guides large-scale navigation, consistent with a role for visual imagery or a cognitive map.

Beyond visual, aural and haptic movement perception: hMT+ is activated by electrotactile motion stimulation of the tongue in sighted and in congenitally blind individuals

The motion-sensitive middle temporal cortex (hMT+ complex) responds also to non-visual motion stimulation conveyed through the tactile and auditory modalities, both in sighted and in congenitally blind individuals. This indicates that hMT+ is truly responsive to motion-related information regardless of visual experience and the sensory modality through which such information is carried to the brain. Here we determined whether the hMT+ complex responds to motion perception per se, that is, motion not perceived through the visual, haptic or aural modalities. Using functional magnetic resonance imaging (fMRI), we investigated brain responses in eight congenitally blind and nine sighted volunteers who had been trained to use the tongue display unit (TDU), a sensory substitution device which converts visual information into electrotactile pulses delivered to the tongue, to resolve a tactile motion discrimination task. Stimuli consisted of either static dots, dots moving coherently or dots moving in random directions. Both groups learned the task at the same rate and activated the hMT+ complex during tactile motion discrimination, although at different anatomical locations. Furthermore, the congenitally blind subjects showed additional activations within the dorsal extrastriate cortical pathway. These results extend previous data in support of the supramodal functional organization of hMT+ complex by showing that this cortical area processes motion-related information per se, that is, motion stimuli that are not visual in nature and that are administered to body structures that, in humans, are not primarily devoted to movement perception or spatial location, such as the tongue. In line with previous studies, the differential activations between sighted and congenitally blind individuals indicate that lack of vision leads to functional rearrangements of these supramodal cortical areas.

A real-time adaptive filtering approach to motion artefacts removal from ECG signals

Motion artefacts represent a severe problem in Electrocardiogram (ECG) monitoring using portable devices, as they may overlap the characteristic ECG waveforms. In this study, an adaptive filtering technique for artefacts removal is investigated. The adopted approach exploits motion-related information coming from an accelerometer attached on the ECG electrode. The experimental results indicate that adaptive filtering technique removes a large amount of motion artefacts with a decreased number of beat detection errors, making our approach a useful pre-processing scheme for ECG analysis. The developed system, implemented on a commercial microcontroller, verified real-time requirements, hence resulting suitably for portable monitoring devices.

Patch-Based Video Processing: A Variational Bayesian Approach

In this paper, we present a patch-based variational Bayesian framework for video processing and demonstrate its potential in denoising, inpainting and deinterlacing. Unlike previous methods based on explicit motion estimation, we propose to embed motion-related information into the relationship among video patches and develop a nonlocal sparsity-based prior for typical video sequences. Specifically, we first extend block matching (nearest neighbor search) into patch clustering (k-nearest-neighbor search), which represents motion in an implicit and distributed fashion. Then we show how to exploit the sparsity constraint by sorting and packing similar patches, which can be better understood from a manifold perspective. Under the Bayesian framework, we treat both patch clustering result and unobservable data as latent variables and solve the inference problem via variational EM algorithms. A weighted averaging strategy of fusing diverse inference results from overlapped patches is also developed. The effectiveness of patch-based video models is demonstrated by extensive experimental results on a wide range of video materials.

Impaired visual recognition of biological motion in schizophrenia

Background Motion perception deficits have been suggested to be an important feature of schizophrenia but the behavioral consequences of such deficits are unknown. Biological motion refers to the movements generated by living beings. The human visual system rapidly and effortlessly detects and extracts socially relevant information from biological motion. A deficit in biological motion perception may have significant consequences for detecting and interpreting social information. Methods Schizophrenia patients and matched healthy controls were tested on two visual tasks: recognition of human activity portrayed in point–light animations (biological motion task) and a perceptual control task involving detection of a grouped figure against the background noise (global-form task). Both tasks required detection of a global form against background noise but only the biological motion task required the extraction of motion-related information. Results Schizophrenia patients performed as well as the controls in the global-form task, but were significantly impaired on the biological motion task. In addition, deficits in biological motion perception correlated with impaired social functioning as measured by the Zigler social competence scale [Zigler, E., Levine, J. (1981). Premorbid competence in schizophrenia: what is being measured? Journal of Consulting and Clinical Psychology, 49, 96–105.]. Conclusion The deficit in biological motion processing, which may be related to the previously documented deficit in global motion processing, could contribute to abnormal social functioning in schizophrenia.

Multiple reference frames for motion in the primate cerebellum.

Knowledge of body motion through space is necessary for spatial orientation, self-motion perception, and postural control. Yet, sensory afferent signals may not directly provide such information to the brain. Because motion detected by the vestibular end organs is encoded in a head-fixed frame of reference, a coordinate transformation is thus required to encode body motion. In this study, we investigated whether cerebellar motion-sensitive neurons encode the translation of the body through space. We systematically changed both the direction of motion relative to the body and the static orientation of the head relative to the trunk. The activities of motion-sensitive neurons in the most medial of the deep cerebellar nuclei, the rostral fastigial nucleus, were compared with those in the brainstem vestibular nuclei. We found a distributed representation of reference frames for motion in the rostral fastigial nucleus, in contrast to cells in the vestibular nuclei, which primarily encoded motion in a head-fixed reference frame. This differential representation of motion-related information implies potential differences in the functional roles of these areas.