Depth Motion Research Articles

Interocular velocity cues elicit vergence eye movements in mice.

We stabilize the dynamic visual world on our retina by moving our eyes in response to motion signals. Coordinated movements between the two eyes are characterized as version when both eyes move in the same direction and vergence when the two eyes move in opposite directions. Vergence eye movements are necessary to track objects in three dimensions. In primates they can be elicited by intraocular differences in either spatial signals (disparity) or velocity, requiring the integration of left and right eye inputs. Whether mice are capable of similar behaviors is not known. To address this issue, we measured vergence eye movements in mice using a stereoscopic stimulus known to elicit vergence eye movements in primates. We found that mice also exhibit vergence eye movements, although at a low gain and that the primary driver of these vergence eye movements is interocular motion. Spatial disparity cues alone are ineffective. We also found that the vergence eye movements we observed in mice were robust to silencing visual cortex and to manipulations that disrupt the normal development of binocularity in visual cortex. A sublinear combination of motor commands driven by monocular signals is sufficient to account for our results.NEW & NOTEWORTHY The visual system integrates signals from the left and right eye to generate a representation of the world in depth. The binocular integration of signals may be observed from the coordinated vergence eye movements elicited by object motion in depth. We explored the circuits and signals responsible for these vergence eye movements in rodent and find these vergence eye movements are generated by a comparison of the motion and not spatial visual signals.

Three-Dimensional Motion Perception: Comparing Speed and Speed Change Discrimination for Looming Stimuli.

Judging the speed of objects moving in three dimensions is important in our everyday lives because we interact with objects in a three-dimensional world. However, speed perception has been seldom studied for motion in depth, particularly when using monocular cues such as looming. Here, we compared speed discrimination, and speed change discrimination, for looming stimuli, in order to better understand what visual information is used for these tasks. For the speed discrimination task, we manipulated the distance and duration information available, in order to investigate if participants were specifically using speed information. For speed change discrimination, total distance and duration were held constant; hence, they could not be used to successfully perform that task. For the speed change discrimination task, our data were consistent with observers not responding specifically to speed changes within an interval. Instead, they may have used alternative, arguably less optimal, strategies to complete the task. Evidence suggested that participants used a variety of cues to complete the speed discrimination task, not always solely relying on speed. Further, our data suggested that participants may have switched between cues on a trial to trial basis. We conclude that speed changes in looming stimuli were not used in a speed change discrimination task, and that naïve participants may not always exclusively use speed for speed discrimination.

A motion-in-depth model based on inter-ocular velocity to estimate direction in depth

Perception of motion in depth is one of the most important visual functions for living in the three-dimensional world. Two binocular cues have been investigated for motion in depth: inter-ocular velocity difference (IOVD) and changing disparity (CD). IOVD provides direction information directly by comparing velocity signals from the two retinas. In this study, we propose for the first time a motion-in-depth model of IOVD that predicts motion-in-depth direction. The model is based on a psychophysical assumption that there are four channels tuned to different directions in depth (Journal of Physiology 235 (1973) 17–29). We modeled these channels by combining outputs of low-level motion detectors that are sensitive to left and right retinal stimulation. Using these channels, we constructed a model of motion in depth that successfully predicted a variety of psychophysical results including direction discrimination, perceived direction, spatial frequency tuning, effect of speed on rotation in depth, effect of lateral motion direction, and effect of binocular and temporal correlations.

Self Intelligence with Human Activities Recognition based on Convolutional Neural Networks

In the presented paper, we propose a strategy related to activity recognition of human from profundity maps as well as sequences stance information using convolutional neural systems. Two information descriptors will be utilized for activity portrayal. The main information is a depth movement picture which will store back to back depth motion images of a human activity, whilst the subsequent data is the proposed moving joint description feature which conveys the movement of joints after time instants. To boost highlight extraction for precise activity arrangement, we will use three networked channels prepared with different inputs along with hypothesis verification. The activity results produced from those channels are intertwined for last activity characterization. Here, we suggest a few combination score based tasks to amplify the weightage of the correct activity. The experiments reveal the aftereffects of intertwining the yield of those channels along with the hypothesis are superior to utilizing a single channel or intertwining more than one channel in particular. The technique was assessed on two open databases which are Microsoft activity dataset and the second one is taken from University of Texas . The results demonstrate that our method beats the vast majority of existing cutting edge techniques, for example, histogram of arranged 4-D normal in datasets. Albeit DHA dataset has high number of activities (38 activities) contrasted with existing activity datasets, our paper outperforms a cutting edge strategy on the dataset by 6.9%.

Multi-view region-adaptive multi-temporal DMM and RGB action recognition

Human action recognition remains an important yet challenging task. This work proposes a novel action recognition system. It uses a novel multi-view region-adaptive multi-resolution-in-time depth motion map (MV-RAMDMM) formulation combined with appearance information. Multi-stream 3D convolutional neural networks (CNNs) are trained on the different views and time resolutions of the region-adaptive depth motion maps. Multiple views are synthesised to enhance the view invariance. The region-adaptive weights, based on localised motion, accentuate and differentiate parts of actions possessing faster motion. Dedicated 3D CNN streams for multi-time resolution appearance information are also included. These help to identify and differentiate between small object interactions. A pre-trained 3D-CNN is used here with fine-tuning for each stream along with multi-class support vector machines. Average score fusion is used on the output. The developed approach is capable of recognising both human action and human–object interaction. Three public-domain data-sets, namely MSR 3D Action, Northwestern UCLA multi-view actions and MSR 3D daily activity, are used to evaluate the proposed solution. The experimental results demonstrate the robustness of this approach compared with state-of-the-art algorithms.

Cue-dependent effects of VR experience on motion-in-depth sensitivity.

The visual system exploits multiple signals, including monocular and binocular cues, to determine the motion of objects through depth. In the laboratory, sensitivity to different three-dimensional (3D) motion cues varies across observers and is often weak for binocular cues. However, laboratory assessments may reflect factors beyond inherent perceptual sensitivity. For example, the appearance of weak binocular sensitivity may relate to extensive prior experience with two-dimensional (2D) displays in which binocular cues are not informative. Here we evaluated the impact of experience on motion-in-depth (MID) sensitivity in a virtual reality (VR) environment. We tested a large cohort of observers who reported having no prior VR experience and found that binocular cue sensitivity was substantially weaker than monocular cue sensitivity. As expected, sensitivity was greater when monocular and binocular cues were presented together than in isolation. Surprisingly, the addition of motion parallax signals appeared to cause observers to rely almost exclusively on monocular cues. As observers gained experience in the VR task, sensitivity to monocular and binocular cues increased. Notably, most observers were unable to distinguish the direction of MID based on binocular cues above chance level when tested early in the experiment, whereas most showed statistically significant sensitivity to binocular cues when tested late in the experiment. This result suggests that observers may discount binocular cues when they are first encountered in a VR environment. Laboratory assessments may thus underestimate the sensitivity of inexperienced observers to MID, especially for binocular cues.

Privacy-preserving human action recognition as a remote cloud service using RGB-D sensors and deep CNN

Cloud-based expert systems are highly emerging nowadays. However, the data owners and cloud service providers are not in the same trusted domain in practice. For the sake of data privacy, sensitive data usually has to be encrypted before outsourcing which makes effective cloud utilization a challenging task. Taking this concern into account, we propose a novel cloud-based approach to securely recognize human activities. A few schemes exist in the literature for secure recognition. However, they suffer from the problem of constrained data and are vulnerable to re-identification attack, where advanced deep learning models are used to predict an object’s identity. We address these problems by considering color and depth data, and securing them using position based superpixel transformation. The proposed transformation is designed by actively involving additional noise while resizing the underlying image. Due to this, a higher degree of obfuscation is achieved. Further, in spite of securing the complete video, we secure only four images, that is, one motion history image and three depth motion maps which are highly saving the data overhead. The recognition is performed using a four stream deep Convolutional Neural Network (CNN), where each stream is based on pre-trained MobileNet architecture. Experimental results show that the proposed approach is the best suitable candidate in “security-recognition accuracy (%)” trade-off relation among other image obfuscation as well as state-of-the-art schemes. Moreover, a number of security tests and analyses demonstrate robustness of the proposed approach.

A Flexible High-Level Fusion for an Accurate Human Action Recognition System

Action recognition is a very effective method of computer vision areas. In the last few years, there has been a growing interest in Deep learning networks as the Long Short–Term Memory (LSTM) architectures due to their efficiency in long-term time sequence processing. In the light of these recent events in deep neural networks, there is now considerable concern about the development of an accurate action recognition approach with low complexity. This paper aims to introduce a method for learning depth activity videos based on the LSTM and the classification fusion. The first step consists in extracting compact depth video features. We start with the calculation of Depth Motion Maps (DMM) from each sequence. Then we encode and concatenate contour and texture DMM characteristics using the histogram-of-oriented-gradient and local-binary-patterns descriptors. The second step is the depth video classification based on the naive Bayes fusion approach. Training three classifiers, which are the collaborative representation classifier, the kernel-based extreme learning machine and the LSTM, is done separately to get classification scores. Finally, we fuse the classification score outputs of all classifiers with the naive Bayesian method to get a final predicted label. Our proposed method achieves a significant improvement in the recognition rate compared to previous work that has used Kinect v2 and UTD-MHAD human action datasets.

Multi-Modal Human Action Recognition With Sub-Action Exploiting and Class-Privacy Preserved Collaborative Representation Learning

Multimodal human action recognition with depth sensors has drawn wide attention, due to its potential applications such as health-care monitoring, smart buildings/home, intelligent transportation, and security surveillance. As one of the obstacles of robust action recognition, sub-actions sharing, especially among similar action categories, makes human action recognition more challenging. This paper proposes a segmental architecture to exploit the relations of sub-actions, jointly with heterogeneous information fusion and Class-privacy Preserved Collaborative Representation (CPPCR) for multi-modal human action recognition. Specifically, a segmental architecture is proposed based on the normalized action motion energy. It models long-range temporal structure over video sequences to better distinguish the similar actions bearing sub-action sharing phenomenon. The sub-action based depth motion and skeleton features are then extracted and fused. Moreover, by introducing within-class local consistency into Collaborative Representation (CR) coding, CPPCR is proposed to address the challenging sub-action sharing phenomenon, learning the high-level discriminative representation. Experiments on four datasets demonstrate the effectiveness of the proposed method.

Convergence and divergence to radial optic flow in infancy.

Research finds a relationship between the development of depth perception and ocular motion functions including smooth pursuit and ocular following response. Infants' reactions to looming stimuli also suggest sensitivity to optic flow information that specifies relative distance. With radial optic flow, an expanding flow field elicits involuntary convergent eye movements while a contracting one elicits involuntary divergent eye movements. This response suggests the visual system is interpreting the radial flow as a change in relative depth. We measured the oculomotor response to radial optic flow in infants aged two to five months. The stimulus comprised a radial optic flow pattern that expanded or contracted across eight 400 ms trials while eye position was monitored with a Tobii X120 eye tracker. A subset of infants also viewed trials of a static version of the stimulus. On average, most infants in each age group demonstrated convergence to the expanding pattern and divergence to the contracting one. Moreover, the difference in gain between the convergence and divergence eye movements was significant. The presence of correct-direction vergence eye movements in response to expansion and contraction provides further evidence that infants are sensitive to information that specifies relative motion in depth.

Autonomous underwater vehicle depth control based on an improved active disturbance rejection controller

Large fluctuation, large overshoot, and uncertain external disturbance that occur when an autonomous underwater vehicle is in deep motion are difficult to address using the traditional control method. An optimal control strategy based on an improved active disturbance rejection control technology is proposed to enhance the trajectory tracking accuracy of autonomous underwater vehicles in actual bathymetric operations and resist external and internal disturbances. First, the depth motion and mathematical models of an autonomous underwater vehicle and propeller are established, respectively. Second, the control rate of the extended state observer and the nonlinear error feedback of the traditional active disturbance rejection control are improved by using a new nonlinear function. The nonlinearity, model uncertainty, and external disturbance of the autonomous underwater vehicle depth control system are extended to a new state, which is realized by an improved extended state observer. Third, the improved nonlinear state error feedback is used to suppress residual errors and provide high-quality control for the system. Simulation and experimental results show that under the same parameters, the traditional active disturbance rejection control has a small overshoot, fast tracking ability, and strong anti-interference ability. The optimized active disturbance rejection control and traditional active disturbance rejection control are applied to the deep-variation motion of autonomous underwater vehicles. Results show that the proposed optimal control strategy is not only simple and feasible but also demonstrates good control performance.

Object kinetics perception in auto-stereoscopic vision.

In this paper, we study the conditions for the perception of object kinetics, produced by a multiview auto-stereoscopic system and a set of still images. We assess the capabilities and performances of such an optical system to encode complex trajectories and kinetics of objects moving in depth. In particular, we set up rules to create motions with nonuniform velocities, when obtained by motion parallax induced by the observer. We establish a link with plenoptic systems from where we derive applicable scaling rules to ensure stereoscopic vision and to provide fluid motion perception with satisfying visual comfort. Finally, we scale the optical system, thanks to obtained parameters, to emulate the perception of object motions in depth with fluid kinetics and to create impressive motion effects.

Misperception of motion in depth originates from an incomplete transformation of retinal signals.

Depth perception requires the use of an internal model of the eye-head geometry to infer distance from binocular retinal images and extraretinal 3D eye-head information, particularly ocular vergence. Similarly, for motion in depth perception, gaze angle is required to correctly interpret the spatial direction of motion from retinal images; however, it is unknown whether the brain can make adequate use of extraretinal version and vergence information to correctly transform binocular retinal motion into 3D spatial coordinates. Here we tested this hypothesis by asking participants to reconstruct the spatial trajectory of an isolated disparity stimulus moving in depth either peri-foveally or peripherally while participants' gaze was oriented at different vergence and version angles. We found large systematic errors in the perceived motion trajectory that reflected an intermediate reference frame between a purely retinal interpretation of binocular retinal motion (not accounting for veridical vergence and version) and the spatially correct motion. We quantify these errors with a 3D reference frame model accounting for target, eye, and head position upon motion percept encoding. This model could capture the behavior well, revealing that participants tended to underestimate their version by up to 17%, overestimate their vergence by up to 22%, and underestimate the overall change in retinal disparity by up to 64%, and that the use of extraretinal information depended on retinal eccentricity. Since such large perceptual errors are not observed in everyday viewing, we suggest that both monocular retinal cues and binocular extraretinal signals are required for accurate real-world motion in depth perception.

Deep Learning of Fuzzy Weighted Multi-Resolution Depth Motion Maps with Spatial Feature Fusion for Action Recognition.

Human action recognition (HAR) is an important yet challenging task. This paper presents a novel method. First, fuzzy weight functions are used in computations of depth motion maps (DMMs). Multiple length motion information is also used. These features are referred to as fuzzy weighted multi-resolution DMMs (FWMDMMs). This formulation allows for various aspects of individual actions to be emphasized. It also helps to characterise the importance of the temporal dimension. This is important to help overcome, e.g., variations in time over which a single type of action might be performed. A deep convolutional neural network (CNN) motion model is created and trained to extract discriminative and compact features. Transfer learning is also used to extract spatial information from RGB and depth data using the AlexNet network. Different late fusion techniques are then investigated to fuse the deep motion model with the spatial network. The result is a spatial temporal HAR model. The developed approach is capable of recognising both human action and human–object interaction. Three public domain datasets are used to evaluate the proposed solution. The experimental results demonstrate the robustness of this approach compared with state-of-the art algorithms.

Encoding- and decision-related brain activity during a motion judgment task

We have previously used steady-state VEPs to show that responses in visual cortex are strongly suppressed when motion stimuli produce inter-ocular velocity differences (Kohler et al., 2018), regardless of whether these differences give rise to movement in depth (MID). Here we used an event-related VEP design to explore how activity evoked by lateral movement and MID unfolds over time. Participants (n=20) viewed stereoscopically presented random-dot patterns in which a central region moved in one of four directions. Dots underwent single-shot apparent motion on every trial: in-phase motion between the two eyes produced lateral movement (left/right), while anti-phase motion gave rise to MID (towards/away). Participants indicated movement direction as quickly and accurately as possible using the keyboard arrow keys. For lateral movement, accuracy was at ceiling and reaction time (RT) was ~800ms, while MID produced lower accuracies (~80%) and longer RTs (~1000ms). We used reliable components analysis to extract maximally correlated signal components from the EEG data (Dmochowski et al., 2012). A medio-frontal component appeared to capture decision-related activity, exhibiting a ramp-like shape that was shallower for MID, consistent with a longer integration period leading to slower RTs. An additional component, centered over occipital cortex, appeared to primarily capture encoding-related activity. Lateral movement and MID produced indistinguishable positive peaks at ~70ms, but diverged at later negative (~180ms) and positive (~350ms) peaks. MID responses were enhanced (more negative) around the negative peak (100–200ms) and suppressed (less positive) leading up to the second positive peak (250–350ms). This late-onset suppression is consistent with the finding that suppression depends on second-order processes, including extraction of relative motion and relative disparity (Kohler et al., 2018). The current results provide a time-resolved electrophysiological analogue to psychophysical data showing that temporal integration underlying decision making is near-perfect for lateral motion, but sub-optimal for MID (Katz et al., 2015).

Deep Convolutional Neural Networks for Human Action Recognition Using Depth Maps and Postures

In this paper, we present a method (Action-Fusion) for human action recognition from depth maps and posture data using convolutional neural networks (CNNs). Two input descriptors are used for action representation. The first input is a depth motion image that accumulates consecutive depth maps of a human action, whilst the second input is a proposed moving joints descriptor which represents the motion of body joints over time. In order to maximize feature extraction for accurate action classification, three CNN channels are trained with different inputs. The first channel is trained with depth motion images (DMIs), the second channel is trained with both DMIs and moving joint descriptors together, and the third channel is trained with moving joint descriptors only. The action predictions generated from the three CNN channels are fused together for the final action classification. We propose several fusion score operations to maximize the score of the right action. The experiments show that the results of fusing the output of three channels are better than using one channel or fusing two channels only. Our proposed method was evaluated on three public datasets: 1) Microsoft action 3-D dataset (MSRAction3D); 2) University of Texas at Dallas-multimodal human action dataset; and 3) multimodal action dataset (MAD) dataset. The testing results indicate that the proposed approach outperforms most of existing state-of-the-art methods, such as histogram of oriented 4-D normals and Actionlet on MSRAction3D. Although MAD dataset contains a high number of actions (35 actions) compared to existing action RGB-D datasets, this paper surpasses a state-of-the-art method on the dataset by 6.84%.

Asymmetries between achromatic and chromatic extraction of 3D motion signals

Motion in depth (MID) can be cued by high-resolution changes in binocular disparity over time (CD), and low-resolution interocular velocity differences (IOVD). Computational differences between these two mechanisms suggest that they may be implemented in visual pathways with different spatial and temporal resolutions. Here, we used fMRI to examine how achromatic and S-cone signals contribute to human MID perception. Both CD and IOVD stimuli evoked responses in a widespread network that included early visual areas, parts of the dorsal and ventral streams, and motion-selective area hMT+. Crucially, however, we measured an interaction between MID type and chromaticity. fMRI CD responses were largely driven by achromatic stimuli, but IOVD responses were better driven by isoluminant S-cone inputs. In our psychophysical experiments, when S-cone and achromatic stimuli were matched for perceived contrast, participants were equally sensitive to the MID in achromatic and S-cone IOVD stimuli. In comparison, they were relatively insensitive to S-cone CD. These findings provide evidence that MID mechanisms asymmetrically draw on information in precortical pathways. An early opponent motion signal optimally conveyed by the S-cone pathway may provide a substantial contribution to the IOVD mechanism.

Dynamic 3D Hand Gesture Recognition by Learning Weighted Depth Motion Maps

Hand gesture recognition (HGR) from sequences of depth maps is a challenging computer vision task because of the low inter-class and high intra-class variability, different execution rates of each gesture, and the high articulated nature of the human hand. In this paper, a multilevel temporal sampling (MTS) method is first proposed that is based on the motion energy of keyframes of depth sequences. As a result, long, middle, and short sequences are generated that contain the relevant gesture information. The MTS results in increasing the intra-class similarity while raising the inter-class dissimilarities. The weighted depth motion map (WDMM) is then proposed to extract the spatiotemporal information from generated summarized sequences by an accumulated weighted absolute difference of consecutive frames. The histogram of gradient and local binary pattern are exploited to extract features from WDMM. The obtained results define the current state-of-the-art on three public benchmark datasets of: MSR Gesture 3D, SKIG, and MSR Action 3D, for 3D HGR. We also achieve competitive results on NTU action dataset.

ACTION RECOGNITION USING UNDECIMATED DUAL TREE COMPLEX WAVELET TRANSFORM FROM DEPTH MOTION MAPS / DEPTH SEQUENCES

Abstract. Accumulating the motion information from a video sequence is one of the highly challenging and significant phase in Human Action Recognition. To achieve this, several classical and compact representations are proposed by the research community with proven applicability. In this paper, we propose a compact Depth Motion Map based representation methodology with hastey striding, consisely accumulating the motion information. We extract Undecimated Dual Tree Complex Wavelet Transform features from the proposed DMM, to form an efficient feature descriptor. We designate a Sequential Extreme Learning Machine for classifying the human action secquences on benchmark datasets, MSR Action 3D dataset and DHA Dataset. We empirically prove the feasability of our method under standard protocols, achieving proven results.

Speed change discrimination for motion in depth using constant world and retinal speeds.

Motion at constant speed in the world maps into retinal motion very differently for lateral motion and motion in depth. The former is close to linear, for the latter, constant speed objects accelerate on the retina as they approach. Motion in depth is frequently studied using speeds that are constant on the retina, and are thus not consistent with real-world constant motion. Our aim here was to test whether this matters: are we more sensitive to real-world motion? We measured speed change discrimination for objects undergoing accelerating retinal motion in depth (consistent with constant real-world speed), and constant retinal motion in depth (consistent with real-world deceleration). Our stimuli contained both looming and binocular disparity cues to motion in depth. We used a speed change discrimination task to obtain thresholds for conditions with and without binocular and looming motion in depth cues. We found that speed change discrimination thresholds were similar for accelerating retinal speed and constant retinal speed and were notably poor compared to classic speed discrimination thresholds. We conclude that the ecologically valid retinal acceleration in our stimuli neither helps, nor hinders, our ability to make judgements in a speed change discrimination task.