Depth Motion Research Articles

Theoretical prerequisites for the bionic substantiation of spring soil leveler working bodies parameters

In the system of measures for pre-sowing tillage, special attention should be given to leveling the soil, ensuring an increase in the uniformity of seeding in depth, which has a significant impact on increasing yields. The main solution to this problem is to improve the design of aligners using a mechanical-bionic approach for the analytical description of the optimal forms and parameters of their working bodies. The design of the working body of the spring soil leveler is proposed, the cutting edges of the undercutting wings of which have ridges and hollows in the form of a logarithmic curve with a positive convexity of curvature inside the depressions, developed by the bionic prototype of the mole cricket (Gryllotalpa). Using the methods of agricultural mechanics, the theoretical dependence of the rational depth of the working bodies and the value of the tractive resistance of spring soil leveler on the travel speed, design parameters and physical-mechanical soil properties, which are proved by the results of experimental research on the south carbonated middle loamy black earth have been established. The experimental spring soil leveler provided the reduction of traction resistance by 7...8% (40...49 N) and had the best stability of motion in depth by an average of 1.5 times (± 0.5 cm) in comparison with the serial model of the working bodies of the leveler VPN-5.6 A.

Perceived speed of motion in depth modulates misjudgements of approaching trajectories consistently with a slow prior

Previous studies have shown that the angle of approach is consistently overestimated for approaching (but passing-by) objects. An explanation based on a slow-motion prior has been proposed in the past to account for this bias. The mechanism relies on the (less reliable) in-depth component of the motion being more attracted towards the slow motion prior than the (more reliable) lateral component. This hypothesis predicts that faster speeds in depth will translate into a greater bias if the perception of velocity in depth follows Weber’s law. Our approach is different than the one used in previous studies where perceived speed and direction were measured in different experiments. To test our hypothesis, we conducted an experiment in which participants estimated approaching angles via a pointing device, while at the same time comparing the speed of the approaching object with a lateral velocity reference. This way, we couple perceived speed with perceived trajectory for each approaching angle in the same trial. Our results show that the directional bias is larger for faster objects, which is consistent with motion in depth following Weber’s law. The differential biases can be accounted for by a Bayesian model that includes a slow motion prior.

Contributions of binocular and monocular cues to motion-in-depth perception.

Intercepting and avoiding moving objects requires accurate motion-in-depth (MID) perception. Such motion can be estimated based on both binocular and monocular cues. Because previous studies largely characterized sensitivity to these cues individually, their relative contributions to MID perception remain unclear. Here we measured sensitivity to binocular, monocular, and combined cue MID stimuli using a motion coherence paradigm. We first confirmed prior reports of substantial variability in binocular MID cue sensitivity across the visual field. The stimuli were matched for eccentricity and speed, suggesting that this variability has a neural basis. Second, we determined that monocular MID cue sensitivity also varied considerably across the visual field. A major component of this variability was geometric: An MID stimulus produces the largest motion signals in the eye contralateral to its visual field location. This resulted in better monocular discrimination performance when the contralateral rather than ipsilateral eye was stimulated. Third, we found that monocular cue sensitivity generally exceeded, and was independent of, binocular cue sensitivity. Finally, contralateral monocular cue sensitivity was found to be a strong predictor of combined cue sensitivity. These results reveal distinct factors constraining the contributions of binocular and monocular cues to three-dimensional motion perception.

Action recognition from depth sequence using depth motion maps-based local ternary patterns and CNN

This paper presents a method for human action recognition from depth sequences captured by the depth camera. The main idea of the method is the action mapping image classification via convolutional neural network (CNN) based approach. Firstly, we project the raw frames onto three orthogonal Cartesian planes and stack the results into three still images (corresponding to the front, side, and top views) to form the Depth Motion Maps (DMMs). Secondly, Local Ternary Pattern (LTP) is introduced as an image filter for DMMs, thus to improve the distinguishability of similar actions. Finally, we apply CNN to action recognition by classifying corresponding LTP-encoded images. Experimental results on the popular and challenging benchmark MSR-Action 3D and MSR-Gesture dataset show the effectiveness of the presented method and meet real-time action recognition task requirements.

A bio-inspired motion sensitive model and its application to estimating human gaze positions under classified driving conditions

Human visual attention performs the spatio-temporal regularity, an inherent regularity in dynamic vision, for the natural visual tasks. However, recent computational visual attention models still have deficiency to reflect the spatio-temporal regularity thoroughly. Motivated by this, we propose a bio-inspired motion sensitive model to estimate human gaze positions in driving. The proposed model has four key advantages. First, inspired by two types of motion sensitive neurons, this model can perceive the motion cues from different directions spatially and temporally. Second, compared to conventional deep learning based models, this model does not rely on expensive training samples with gaze annotations. Third, the proposed model is based upon the visual signal processing without constructing a complex deep neural network architecture, which enables this model to be implemented with low-cost hardware. Fourth, inspired by the visual pathway in drosophila motion vision, the visual signal processing on directional and depth motion sensitive map largely enhances this model's competence in a similar way of human gaze positions in driving. To test this proposed model, we collect a driving scene dataset from the perspective of egocentric vision that aims to systematically evaluate the performance of spatio-temporal visual attentional models. The video clips in the dataset are categorized into ten popular driving conditions. The proposed model is evaluated by comparing with the human baseline of the gaze positions. Experimental results demonstrate that the proposed model can effectively estimate the human gaze positions in driving and consistently outperforms traditional visual attention models as well as the deep learning based model.

Robust Human Activity Recognition Using Multimodal Feature-Level Fusion

Automated recognition of human activities or actions has great significance as it incorporates wide-ranging applications, including surveillance, robotics, and personal health monitoring. Over the past few years, many computer vision-based methods have been developed for recognizing human actions from RGB and depth camera videos. These methods include space-time trajectory, motion encoding, key poses extraction, space-time occupancy patterns, depth motion maps, and skeleton joints. However, these camera-based approaches are affected by background clutter and illumination changes and applicable to a limited field of view only. Wearable inertial sensors provide a viable solution to these challenges but are subject to several limitations such as location and orientation sensitivity. Due to the complementary trait of the data obtained from the camera and inertial sensors, the utilization of multiple sensing modalities for accurate recognition of human actions is gradually increasing. This paper presents a viable multimodal feature-level fusion approach for robust human action recognition, which utilizes data from multiple sensors, including RGB camera, depth sensor, and wearable inertial sensors. We extracted the computationally efficient features from the data obtained from RGB-D video camera and inertial body sensors. These features include densely extracted histogram of oriented gradient (HOG) features from RGB/depth videos and statistical signal attributes from wearable sensors data. The proposed human action recognition (HAR) framework is tested on a publicly available multimodal human action dataset UTD-MHAD consisting of 27 different human actions. K-nearest neighbor and support vector machine classifiers are used for training and testing the proposed fusion model for HAR. The experimental results indicate that the proposed scheme achieves better recognition results as compared to the state of the art. The feature-level fusion of RGB and inertial sensors provides the overall best performance for the proposed system, with an accuracy rate of 97.6%.

Multimodal Spatiotemporal Networks for Sign Language Recognition

Different from other human behaviors, sign language has the characteristics of limited local motion of upper limb and meticulous hand action. Some sign language gestures are ambiguous in RGB video due to the influence of lighting and background color, which affects the recognition accuracy. We propose a multimodal deep learning architecture for sign language recognition which effectively combines RGB-D input and two-stream spatiotemporal networks. Depth videos, as an effective compensation of RGB input, can supply additional distance information about the signer's hands. A novel sampling method called ARSS (Aligned Random Sampling in Segments) is put forward to select and align optimal RGB-D video frames, which improves the capacity utilization of multimodal data and reduces the redundancy. We get the hand ROI by joints information of RGB data for local focus in spatial stream. D-shift Net is proposed as depth motion feature extraction in temporal stream, which fully utilizes three dimensional motion information of the sign language. Both streams are fused by convolutional fusion layer to get complementary features. Our approach explored the multimodal information and enhanced the recognition precision. It obtains the state-the-of-art performance on the datasets of CSL (96.7%) and IsoGD (63.78%).

Real-time human action recognition using depth motion maps and convolutional neural networks

Real-time human action recognition using depth motion maps and convolutional neural networks

Combining Adaptive Hierarchical Depth Motion Maps With Skeletal Joints for Human Action Recognition

This paper presents a new framework for human action recognition by fusing human motion with skeletal joints. First, adaptive hierarchical depth motion maps (AH-DMMs) are proposed to capture the shape and motion cues of action sequences. Specifically, AH-DMMs are calculated over adaptive hierarchical windows and Gabor filters are used to encode the texture information of AH-DMMs. Then, spatial distances of skeletal joint positions are computed to characterize the structure information of the human body. Finally, two types of fusion methods including feature-level fusion and decision-level fusion are employed to combine the motion cues and structure information. The experimental results on public benchmark datasets, i.e., MSRAction3D and UTKinect-Action, show the effectiveness of the proposed method.

Depth‐based end‐to‐end deep network for human action recognition

Recognition of human actions from videos can be improved if depth information is available. Depth information certainly helps in segregating foreground motion from the background. Single image depth estimation (SIDE) is a commonly used method for the analysis of weather degraded images. In this study, the idea of SIDE is extended to human action recognition (HAR) on datasets where depth information is not available. Several depth‐based HAR algorithms are available but all of them are using the depth information given with the dataset. Some other methods are using depth motion map which refers to the depth of motion in a temporal direction. Here, a new depth‐based end‐to‐end deep network is proposed for HAR in which the frame‐wise depth is estimated and this estimated depth is used for processing instead of RGB frame. As colour information is not required for estimating motion, a single channel depth map is used for estimating motion in the video. It makes the system computationally efficient. The proposed method is tested and verified on three benchmark datasets namely JHMDB, HMDB51 and UCF101. The proposed method outperforms the existing state‐of‐the‐art methods for HAR on all the three tested datasets.

Action recognition using multi-directional projected depth motion maps

Camera-based action recognition plays a key role in diverse computer vision applications such as human computer interaction. This paper proposes a new action recognition approach using multi-directional projected depth motion map based motion descriptors. First, for the input depth video sequence, all the depth frames in the video are projected onto multiple planes to form the projected images. The absolute difference between two consecutive projected images is accumulated through the entire depth video for establishing maps from multiple views. Then, the local motion consistency of the map is examined to form a histogram of local binary patterns, which are then concatenated and further incorporated into a kernel-based extreme learning machine for action recognition. In contrast to that only three directions are used to calculated the projected depth images for motion feature extraction in the conventional approaches, the proposed approach is able to provide an effective and flexible framework to examine the depth motion maps in multiple projected directions. The proposed approach is evaluated in the well-known MSRA action and gesture video benchmark datasets to demonstrate its superior performance.

Action Recognition Based on Depth Motion Map and Hybrid Classifier

In order to efficiently extract and encode 3D information of human action from depth images, we present a feature extraction and recognition method based on depth video sequences. First, depth images are projected continuously onto three planes of Cartesian coordinate system, and differential images of the respective projection surfaces are accumulated to obtain the complete 3D information of the depth motion maps (DMMs). Then, discriminative completed LBP (disCLBP) encodes depth motion maps to extract effective human action information. A hybrid classifier combined with Extreme Learning Machine (ELM) and collaborative representation classification (CRC) is employed to reduce the computational complexity while reducing the impact of noise. The proposed method is tested on the MSR-Action3D database; the experimental results show that it achieves 96.0% accuracy and well performs better robustness comparing to other popular approaches.

Searching Human Action Recognition Accuracy from Depth Video Sequences Using HOG and PHOG Shape Features

Research on human action recognition from depth video sequences are increasing day by day due to its vast application in automatic surveillance systems, entertainment environments, and healthcare systems etc. In our project, we improve human action recognition accuracy using shape features. We use Histogram of oriented gradients (HOG) and Pyramid Histogram of oriented gradients ( PHOG) to extract shape features. The feature extraction algorithms are used to extract shape feature from dataset of different action videos. At first, depth motion maps (DMMs) are constructed from every action video. Then, the HOG and PHOG features are extracted from each DMMs. Using these features, actions are recognized by the

3D Human Action Recognition Using a Single Depth Feature and Locality-Constrained Affine Subspace Coding

This paper addresses the problem of recognizing human actions from depth videos. We propose a depth-based local descriptor and affine subspace coding representation with locality-constrained affine subspace coding (LASC) for 3D action recognition. First, each depth video sequence is divided into a set of subsequences (i.e., multi-scale sub-actions) based on the normalized motion energy vector. Next, depth motion map-based gradient local auto-correlation features are employed to capture the shape information and motion cues of each sub-action. In order to obtain discriminative and compact representation, we extract the local high-order information of the depth video using LASC. Through experiments, we show that the use of LASC exhibits better performance compared with existing methods such as locality-constrained linear coding. We compared LASC with the state-of-the-art methods based on similar principle, using features extracted from a single modality, on four datasets, and with those using multiple features or nonlinear recognition machines. The results on four datasets clearly show the effectiveness of the proposed method.

Dynamic hand gesture recognition using motion pattern and shape descriptors

The key problems of dynamic hand gesture recognition are large intra-class (gesture types, without considering hand configuration) spatial-temporal variability and similar inter-class (gesture types, only considering hand configuration) motion pattern. Firstly, for intra-class spatial-temporal variability, the key is to reduce the spatial-temporal variability. Due to the average operation can improve the robustness very well, we propose a motion pattern descriptor, Time-Wise Histograms of Oriented Gradients (TWHOG), which extracts the average spatial-temporal information in the space-time domain from three orthogonal projection views (XY, YT, XT). Secondly, for similar inter-class motion pattern, accurate representation of hand configuration is especially important. Therefore, the difference in detail needs to be fully captured, and the shape descriptor can amplify subtle differences. Specifically, we introduce Depth Motion Maps-based Histograms of Oriented Gradients (DMM-HOG) to capture subtle differences in hand configurations between different types of gestures with similar motion patterns. Finally, we concatenate TWHOG and DMM-HOG to form the final feature vector Time-Shape Histograms of Oriented Gradients (TSHOG) and verify the effectiveness of the connection from quantitative and qualitative perspective. Comparison study with the state-of-the-art approaches are conducted on two challenge depth gesture datasets (MSRGesture3D, SKIG). The experiment result shows that TSHOG can achieve satisfactory performance while keeping a relative simple model with lower complexity as well as higher generality.

Sensitivity to Velocity- and Disparity-Based Cues to Motion-In-Depth With and Without Spared Stereopsis in Binocular Visual Impairment.

Two binocular sources of information serve motion-in-depth (MID) perception: changes in disparity over time (CD), and interocular velocity differences (IOVD). While CD requires the computation of small spatial disparities, IOVD could be computed from a much lower-resolution signal. IOVD signals therefore might still be available under conditions of binocular vision impairment (BVI) with limited or no stereopsis, for example, amblyopia. Sensitivity to CD and IOVD was measured in adults who had undergone therapy to correct optical misalignment or amblyopia in childhood (n = 16), as well as normal vision controls with good stereoacuity (n = 8). Observers discriminated the interval containing a smoothly oscillating MID "test" stimulus from a "control" stimulus in a two-interval forced choice paradigm. Of the BVI observers with no static stereoacuity (n = 9), one displayed evidence for sensitivity to IOVD only, while there was otherwise no sensitivity for either CD or IOVD in the group. Generally, BVI observers with measurable stereoacuity (n = 7) displayed a pattern resembling the control group: showing a similar sensitivity for both cues. A neutral density filter placed in front of the fixing eye in a subset of BVI observers did not improve performance. In one BVI observer there was preserved sensitivity to IOVD but not CD, though overall only those BVI observers with at least gross stereopsis were able to detect disparity- or velocity-based cues to MID. The results imply that these logically distinct information sources are somehow coupled, and in some cases BVI observers with no stereopsis may still retain sensitivity to IOVD.

Perception of Looming Motion in Virtual Reality Egocentric Interception Tasks.

Motion in depth is commonly misperceived in Virtual Reality (VR), making it difficult to intercept moving objects, for example, in games. We investigate whether motion cues could be modified to improve these interactions in VR. We developed a time-to-contact estimation task, in which observers ($n=18$n=18) had to indicate by button press when a looming virtual object would collide with their head. We show that users consistently underestimate speed. We construct a user-specific model of motion-in-depth perception, and use this model to propose a novel method to modify monocular depth cues tailored to the specific user, correcting individual response errors in speed estimation. A user study was conducted in a simulated baseball environment and observers were asked to hit a looming baseball back in the direction of the pitcher. The study was conducted with and without intervention and demonstrates the effectiveness of the method in reducing interception errors following cue modifications. The intervention was particularly effective at fast ball speeds where performance is most limited by the user's sensorimotor constraints. The proposed approach is easy to implement and could improve the user experience of interacting with dynamic virtual environments.

Relationship between vection and motion perception in depth.

This study examined the effects of cues to motion in depth - namely, stereoscopic (i.e., changing-disparity cues and interocular velocity differences) and changing-size cues on forward and backward vection. We conducted four experiments in which participants viewed expanding or contracting optical flows with the addition of either or both cues. In Experiment 1, participants reported vection by pressing a button whenever they felt it. After each trial, they also rated the magnitude of the vection (from 0 to 100). In Experiments 2 and 3, the participants rated the perceived velocity and motion-in-depth impression of the flows relative to standard stimuli, respectively. In Experiment 4, the participants rated the perceived depth and distance of the display. We observed enhancements in vection, motion-in-depth impression, and perceived depth and distance when either or both types of cues indicated motion-in-depth, as compared to those when the cues did not (Experiments 1, 3, and 4). The perceived velocity changed with cue conditions only for the high velocity condition (Experiment 2). Correlational analyses showed that the vection can be best explained by the motion-in-depth impression. This was partially supported by the multiple regression analyses. These results indicate that the enhancement of vection caused by cues is related to the impression of motion-in-depth rather than the perceived velocity and perceived three-dimensionality.

Enhancing Motion-In-Depth Perception of Random-Dot Stereograms.

Random-dot stereograms have been widely used to explore the neural mechanisms underlying binocular vision. Although they are a powerful tool to stimulate motion-in-depth (MID) perception, published results report some difficulties in the capacity to perceive MID generated by random-dot stereograms. The purpose of this study was to investigate whether the performance of MID perception could be improved using an appropriate stimulus design. Sixteen inexperienced observers participated in the experiment. A training session was carried out to improve the accuracy of MID detection before the experiment. Four aspects of stimulus design were investigated: presence of a static reference, background texture, relative disparity, and stimulus contrast. Participants' performance in MID direction discrimination was recorded and compared to evaluate whether varying these factors helped MID perception. Results showed that only the presence of background texture had a significant effect on MID direction perception. This study provides suggestions for the design of 3D stimuli in order to facilitate MID perception.

Human action recognition using depth motion maps pyramid and discriminative collaborative representation classifier

Human action recognition is a challenging task in machine learning and pattern recognition. This paper presents an action recognition framework based on depth sequences. An effective feature descriptor named depth motion maps pyramid (DMMP) inspired by DMMs is developed. First, a series of DMMs with temporal scales are constructed to effectively capture spatial–temporal motion patterns of human actions. Then these DMMs are fused to obtain the final descriptor named DMMs pyramid. Second, we propose a discriminative collaborative representation classifier (DCRC), where an extra constraint on the collaborative coefficient is imposed to provide prior knowledge for the representation coefficient. In addition, we apply DCRC to encode the obtained features and recognize the human actions. The proposed framework is evaluated on MSR three-dimensional (3-D) action datasets, MSR hand gesture dataset, UTD-MHAD, and MSR daily Activity3D dataset, respectively. The experimental results indicate the effectiveness of our proposed method for human action recognition.