View-dependent Representations Research Articles

Multi-Resolution 3D Rendering for High-Performance Web AR.

In the context of web augmented reality (AR), 3D rendering that maintains visual quality and frame rate requirements remains a challenge. The lack of a dedicated and efficient 3D format often results in the degraded visual quality of the original data and compromises the user experience. This paper examines the integration of web-streamable view-dependent representations of large-sized and high-resolution 3D models in web AR applications. The developed cross-platform prototype exploits the batched multi-resolution structures of the Nexus.js library as a dedicated lightweight web AR format and tests it against common formats and compression techniques. Built with AR.js and Three.js open-source libraries, it allows the overlay of the multi-resolution models by interactively adjusting the position, rotation and scale parameters. The proposed method includes real-time view-dependent rendering, geometric instancing and 3D pose regression for two types of AR: natural feature tracking (NFT) and location-based positioning for large and textured 3D overlays. The prototype achieves up to a 46% speedup in rendering time compared to optimized glTF models, while a 34 M vertices 3D model is visible in less than 4 s without degraded visual quality in slow 3D networks. The evaluation under various scenes and devices offers insights into how a multi-resolution scheme can be adopted in web AR for high-quality visualization and real-time performance.

Focalized contrastive view-invariant learning for self-supervised skeleton-based action recognition

Learning view-invariant representation is a key to improving feature discrimination power for skeleton-based action recognition. Existing approaches cannot effectively remove the impact of viewpoint due to the implicit view-dependent representations. In this work, we propose a self-supervised framework called Focalized Contrastive View-invariant Learning (FoCoViL), which significantly suppresses the view-specific information on the representation space where the viewpoints are coarsely aligned. By maximizing mutual information with an effective contrastive loss between multi-view sample pairs, FoCoViL associates actions with common view-invariant properties and simultaneously separates the dissimilar ones. We further propose an adaptive focalization method based on pairwise similarity to enhance contrastive learning for a clearer cluster boundary in the learned space. Different from many existing self-supervised representation learning work that rely heavily on supervised classifiers, FoCoViL performs well on both unsupervised and supervised classifiers with superior recognition performance. Extensive experiments also show that the proposed contrastive-based focalization generates a more discriminative latent representation.

The emergence of view-symmetric neural responses to familiar and unfamiliar faces

Successful recognition of familiar faces is thought to depend on the ability to integrate view-dependent representations of a face into a view-invariant representation. It has been proposed that a key intermediate step in achieving view invariance is the representation of symmetrical views. However, key unresolved questions remain, such as whether these representations are specific for naturally occurring changes in viewpoint and whether view-symmetric representations exist for familiar faces. To address these issues, we compared behavioural and neural responses to natural (canonical) and unnatural (noncanonical) rotations of the face. Similarity judgements revealed that symmetrical viewpoints were perceived to be more similar than non-symmetrical viewpoints for both canonical and non-canonical rotations. Next, we measured patterns of neural response from early to higher level regions of visual cortex. Early visual areas showed a view-dependent representation for natural or canonical rotations of the face, such that the similarity between patterns of response were related to the difference in rotation. View symmetric patterns of neural response to canonically rotated faces emerged in higher visual areas, particularly in face-selective regions. The emergence of a view-symmetric representation from a view-dependent representation for canonical rotations of the face was also evident for familiar faces, suggesting that view-symmetry is an important intermediate step in generating view-invariant representations. Finally, we measured neural responses to unnatural or non-canonical rotations of the face. View-symmetric patterns of response were also found in face-selective regions. However, in contrast to natural or canonical rotations of the face, these view-symmetric responses did not arise from an initial view-dependent representation in early visual areas. This suggests differences in the way that view-symmetrical representations emerge with canonical or non-canonical rotations. The similarity in the neural response to canonical views of familiar and unfamiliar faces in the core face network suggests that the neural correlates of familiarity emerge at later stages of processing.

Data-Driven Indoor Scene Modeling from a Single Color Image with Iterative Object Segmentation and Model Retrieval.

We propose a new method for modeling the indoor scene from a single color image. With our system, the user only needs to drag a few semantic bounding boxes surrounding the objects of interest. Our system then automatically finds the most similar 3D models from the ShapeNet model repository and aligns them with the corresponding objects of interest. To achieve this, each 3D model is represented as a group of view-dependent representations generated from a set of synthesized views. We iteratively conduct object segmentation and 3D model retrieval, based on the observation that good segmentation of the objects of interest can significantly improve the accuracy of model retrieval and make it robust to cluttered background and occlusions, and in turn, the retrieved 3D models can be used to assist with object segmentation. Segmentation of all objects of interest is achieved simultaneously under a unified multi-labeling framework which fully utilizes the correspondences between the objects of interest and retrieved model images. Besides, we propose a new method to estimate the scene layout of the input image with the segmentation masks, which helps compose the resulting scene and further improves the modeling result remarkably. We verify the effectiveness of our approach through experimenting with a variety of indoor images and comparing against the relevant methods.

Cross-View Action Recognition Based on Hierarchical View-Shared Dictionary Learning

Recognizing human actions across different views is challenging, since observations of the same action often vary greatly with viewpoints. To solve this problem, most existing methods explore the cross-view feature transfer relationship at video level only, ignoring the sequential composition of action segments therein. In this paper, we propose a novel hierarchical transfer framework, which is based on an action temporal-structure model that contains sequential relationship between action segments at multiple timescales. Thus, it can capture the view invariance of the sequential relationship of segment-level transfer. Additionally, we observe that the original feature distributions under different views differ greatly, leading to view-dependent representations irrelevant to the intrinsic structure of actions. Thus, at each level of the proposed framework, we transform the original feature spaces of different views to a view-shared low-dimensional feature space, and jointly learn a dictionary in this space for these views. This view-shared dictionary captures the common structure of action data across the views and can represent the action segments in a way robust to view changes. Moreover, the proposed method can be kernelized easily, and operate in both unsupervised and supervised cross-view scenarios. Extensive experimental results on the IXMAS and WVU datasets demonstrate superiority of the proposed method over state-of-the-art methods.

The BEVPS: A new test battery to assess visual perceptual and spatial processing abilities in 5–14 year-old children

ABSTRACTThis study aims to examine the different levels of visual perceptual object recognition (early, intermediate, and late) defined in Humphreys and Riddoch’s model as well as basic visual spatial processing in children using a new test battery (BEVPS). It focuses on the age sensitivity, internal coherence, theoretical validity, and convergent validity of this battery. French-speaking, typically developing children (n = 179; 5 to 14 years) were assessed using 15 new computerized subtests. After selecting the most age-sensitive tasks though ceiling effect and correlation analyses, an exploratory factorial analysis was run with the 12 remaining subtests to examine the BEVPS’ theoretical validity. Three separate factors were identified for the assessment of the stimuli’s basic features (F1, four subtests), view-dependent and -independent object representations (F2, six subtests), and basic visual spatial processing (F3, two subtests). Convergent validity analyses revealed positive correlations between F1 and F2 and the Beery-VMI visual perception subtest, while no such correlations were found for F3. Children’s performances progressed until the age of 9–10 years in F1 and in view-independent representations (F2), and until 11–12 years in view-dependent representations (F2). However, no progression with age was observed in F3. Moreover, the selected subtests, present good-to-excellent internal consistency, which indicates that they provide reliable measures for the assessment of visual perceptual processing abilities in children.

Event-Related Potential Effects of Object Repetition Depend on Attention and Part-Whole Configuration.

The effects of spatial attention and part-whole configuration on recognition of repeated objects were investigated with behavioral and event-related potential (ERP) measures. Short-term repetition effects were measured for probe objects as a function of whether a preceding prime object was shown as an intact image or coarsely scrambled (split into two halves) and whether or not it had been attended during the prime display. In line with previous behavioral experiments, priming effects were observed from both intact and split primes for attended objects, but only from intact (repeated same-view) objects when they were unattended. These behavioral results were reflected in ERP waveforms at occipital–temporal locations as more negative-going deflections for repeated items in the time window between 220 and 300 ms after probe onset (N250r). Attended intact images showed generally more enhanced repetition effects than split ones. Unattended images showed repetition effects only when presented in an intact configuration, and this finding was limited to the right-hemisphere electrodes. Repetition effects in earlier (before 200 ms) time windows were limited to attended conditions at occipito-temporal sites during the N1, a component linked to the encoding of object structure, while repetition effects at central locations during the same time window (P150) were found for attended and unattended probes but only when repeated in the same intact configuration. The data indicate that view-generalization is mediated by a combination of analytic (part-based) representations and automatic view-dependent representations.

Trajectory‐based view‐invariant hand gesture recognition by fusing shape and orientation

Traditional studies in vision-based hand gesture recognition remain rooted in view-dependent representations, and hence users are forced to be fronto-parallel to the camera. To solve this problem, view-invariant gesture recognition aims to make the recognition result independent of viewpoint changes. However, in current works the view-invariance is achieved at the price of mixing different gesture patterns that have similar trajectory curve shape but different semantic meanings. For example, the gesture ‘push’ can be mistaken as ‘drag’ from another viewpoint. To address this shortcoming, in this study, the authors use a shape descriptor to extract the view-invariant features of a three-dimensional (3D) trajectory. As the shape features are invariant to omnidirectional viewpoint changes, the orientation features are then added into weight different rotation angles so that similar trajectory shapes are better separated. The proposed method was conducted on two different databases, including a popular Australian Sign Language database and a challenging Kinect Hand Trajectory database. Experimental results show that the proposed algorithm achieves a higher average recognition rate than the state-of-the-art approaches, and can better distinguish confusing gestures while meeting the view-invariant condition.

Like the back of the (right) hand? A new fMRI look on the hand laterality task.

There is a common saying for expressing familiarity with something. It refers to our hands, and strangely enough, in English, one says to know something like the back of the hand, whereas in other cultures, for example, Italy, Spain and France, the same expression is with the palm. Previous behavioural data have suggested that our ability to visually discriminate a right from a left hand is influenced by perspective. This behavioural finding has remained without neurophysiological counterparts. We used an implicit motor imagery task in which 30 right-handed subjects were asked to decide whether a picture portrayed a right rather than a left hand during an fMRI event-related experiment. Both views (back and palm) were used, and the hands were rotated by 45° in 8 possible angles. We replicated previous behavioural evidence by showing faster reaction times for the back-view and view-specific interaction effects with the angle of rotation: for the back view, the longest RTs were with the hand facing down at 180°; for the palm view, the longest RTs were at 90° with the hand pointing away from the midline. In addition, the RTs were particularly faster for back views of the right hand. fMRI measurements revealed a stronger BOLD signal increase in left premotor and parietal cortices for stimuli viewed from the palm, whereas back-view stimuli were associated with stronger occipital activations, suggesting a view-specific cognitive strategy: more visually oriented for the back of the hand; more in need of the support of a motoric imagery process for the palms. Right-hand back views were associated with comparatively smaller BOLD responses, attesting, together with the faster reaction times, to the lesser need for neural labour because of greater familiarity with that view of the hand. These differences suggest the existence of brain-encoded, view-dependent representations of body segments.

View-Based Encoding of Actions in Mirror Neurons of Area F5 in Macaque Premotor Cortex

Converging experimental evidence indicates that mirror neurons in the monkey premotor area F5 encode the goals of observed motor acts [1-3]. However, it is unknown whether they also contribute to encoding the perspective from which the motor acts of others are seen. In order to address this issue, we recorded the visual responses of mirror neurons of monkey area F5 by using a novel experimental paradigm based on the presentation of movies showing grasping motor acts from different visual perspectives. We found that the majority of the tested mirror neurons(74%) exhibited view-dependent activity with responses tuned to specific points of view. A minority of the tested mirror neurons (26%) exhibited view-independent responses. We conclude that view-independent mirror neurons encode action goals irrespective of the details of the observed motor acts, whereas the view-dependent ones might either form an intermediate step in the formation of view independence or contribute to a modulation of view-dependent representations in higher-level visual areas, potentially linking the goals of observed motor acts with their pictorial aspects.

Distinct visual working memory systems for view-dependent and view-invariant representation.

BackgroundHow do people sustain a visual representation of the environment? Currently, many researchers argue that a single visual working memory system sustains non-spatial object information such as colors and shapes. However, previous studies tested visual working memory for two-dimensional objects only. In consequence, the nature of visual working memory for three-dimensional (3D) object representation remains unknown.Methodology/Principal FindingsHere, I show that when sustaining information about 3D objects, visual working memory clearly divides into two separate, specialized memory systems, rather than one system, as was previously thought. One memory system gradually accumulates sensory information, forming an increasingly precise view-dependent representation of the scene over the course of several seconds. A second memory system sustains view-invariant representations of 3D objects. The view-dependent memory system has a storage capacity of 3–4 representations and the view-invariant memory system has a storage capacity of 1–2 representations. These systems can operate independently from one another and do not compete for working memory storage resources.Conclusions/SignificanceThese results provide evidence that visual working memory sustains object information in two separate, specialized memory systems. One memory system sustains view-dependent representations of the scene, akin to the view-specific representations that guide place recognition during navigation in humans, rodents and insects. The second memory system sustains view-invariant representations of 3D objects, akin to the object-based representations that underlie object cognition.

Perceptual learning of view-independence in visuo-haptic object representations

We previously showed that cross-modal recognition of unfamiliar objects is view-independent, in contrast to view-dependence within-modally, in both vision and haptics. Does the view-independent, bisensory representation underlying cross-modal recognition arise from integration of unisensory, view-dependent representations or intermediate, unisensory but view-independent representations? Two psychophysical experiments sought to distinguish between these alternative models. In both experiments, participants began from baseline, within-modal, view-dependence for object recognition in both vision and haptics. The first experiment induced within-modal view-independence by perceptual learning, which was completely and symmetrically transferred cross-modally: visual view-independence acquired through visual learning also resulted in haptic view-independence and vice versa. In the second experiment, both visual and haptic view-dependence were transformed to view-independence by either haptic-visual or visual-haptic cross-modal learning. We conclude that cross-modal view-independence fits with a model in which unisensory view-dependent representations are directly integrated into a bisensory, view-independent representation, rather than via intermediate, unisensory, view-independent representations.

Viewpoint Independent Action Recognition

PDF HTML阅读 XML下载 导出引用 引用提醒 视角无关的动作识别 DOI: 作者: 作者单位: 作者简介: 通讯作者: 中图分类号: 基金项目: Supported by the National Natural Science Foundation of China under Grant No.60673189 (国家自然科学基金) Viewpoint Independent Action Recognition Author: Affiliation: Fund Project: 摘要 | 图/表 | 访问统计 | 参考文献 | 相似文献 | 引证文献 | 资源附件 | 文章评论 摘要:人体动作识别是计算机视觉中一个流行而且重要的研究课题.当观察视角发生变化时,动作识别变得格外困难.至今为止,关于动作识别和手势识别的大多数研究工作都是围绕着视角相关的表达展开的.有一小部分利用了视角不变的表示开展研究,可是它们大多数存在一些缺陷,比如缺少用于识别的足够信息,依赖鲁棒的语义特征点的检测或者是点对应.为了解决这个问题,实现视角无关、动作人无关的动作识别,提出了"包容形状"的表示,这种表示不依赖于特定视角.在人体动作识别中,人的身体旋转通常是引起视角变化的主要原因.包容形状充分利用了两个正交摄像机拍摄的轮廓信息以去除由人的身体旋转产生的影响.从来自两个正交的摄像机拍摄的外轮廓,可以很容易计算得到包容形状.利用包容形状的体态表示和隐马尔可夫模型,取得了非特定人、任意视角下动作识别的很好的实验结果.这些实验结果也表明了包容形状包含有足够区分度的信息.同时提出了包容形状的扩展表示,以便在两个摄像机并不完全正交的更为一般的摄像机配置条件下也可以应用,这极大地加强了其实用价值. Abstract:Action recognition is a popular and important research topic in computer vision. However, it is challenging when facing viewpoint variance. So far, most researches in action recognition remain rooted in view-dependent representations. Some view invariance approaches have been proposed, but most of them suffer from some weaknesses, such as lack of abundant information for recognition, dependency on robust meaningful feature detection or point correspondence. To perform viewpoint and subject independent action recognition, this paper proposes a representation called "Envelop Shape" which is viewpoint insensitive. "Envelop Shape" is easy to acquire from silhouettes using two orthogonal cameras. It makes full use of two cameras' silhouettes to dispel influence caused by human body's vertical rotation, which is often the primary viewpoint variance. With the help of "Envelop Shape" representation and Hidden Markov Model, the inspiring results on action recognition independent of subject and viewpoint are obtained. Results indicate that "Envelop Shape" representation contains enough discriminating features for action recognition. Extension of "Envelop Shape" is also proposed to make it run under fewer restrictions of camera configurations, which increases its application value effectively. 参考文献 相似文献 引证文献

Free viewpoint action recognition using motion history volumes

Action recognition is an important and challenging topic in computer vision , with many important applications including video surveillance, automated cinematography and understanding of social interaction. Yet, most current work in gesture or action interpretation remains rooted in view-dependent representations. This paper introduces Motion History Volumes (MHV) as a free-viewpoint representation for human actions in the case of multiple calibrated, and background-subtracted, video cameras. We present algorithms for computing, aligning and comparing MHVs of different actions performed by different people in a variety of viewpoints. Alignment and comparisons are performed efficiently using Fourier transforms in cylindrical coordinates around the vertical axis. Results indicate that this representation can be used to learn and recognize basic human action classes, independently of gender, body size and viewpoint.

Mixture of view-invariant and view-dependent representations in human object-selective cortex

Mixture of view-invariant and view-dependent representations in human object-selective cortex

Structured but view-dependent representations for visual 3-D object classification

Structured but view-dependent representations for visual 3-D object classification