View Invariance Research Articles

View Invariants for Three-Dimensional Points with Constrained Observer Motion

Images from cameras are a common source of navigation information for a variety of vehicles. Such navigation often requires the matching of observed objects (e.g., landmarks, beacons, stars) in an image to a catalog (or map) of known objects. In many cases, this matching problem is made easier through the use of invariants. However, if the objects are modeled as three-dimensional points in general position, it has long been known that there are no invariants for a camera that is also in general position. This work discusses how invariants are introduced when the camera’s motion is constrained to a line, and proves that this is the only camera path along which invariants are possible. Algorithms are presented for computing both the invariants and the location for a camera undergoing rectilinear motion. The applicability of these ideas is discussed within the context of trains, aircraft, and spacecraft.

The emergence of view-symmetric neural responses to familiar and unfamiliar faces

Successful recognition of familiar faces is thought to depend on the ability to integrate view-dependent representations of a face into a view-invariant representation. It has been proposed that a key intermediate step in achieving view invariance is the representation of symmetrical views. However, key unresolved questions remain, such as whether these representations are specific for naturally occurring changes in viewpoint and whether view-symmetric representations exist for familiar faces. To address these issues, we compared behavioural and neural responses to natural (canonical) and unnatural (noncanonical) rotations of the face. Similarity judgements revealed that symmetrical viewpoints were perceived to be more similar than non-symmetrical viewpoints for both canonical and non-canonical rotations. Next, we measured patterns of neural response from early to higher level regions of visual cortex. Early visual areas showed a view-dependent representation for natural or canonical rotations of the face, such that the similarity between patterns of response were related to the difference in rotation. View symmetric patterns of neural response to canonically rotated faces emerged in higher visual areas, particularly in face-selective regions. The emergence of a view-symmetric representation from a view-dependent representation for canonical rotations of the face was also evident for familiar faces, suggesting that view-symmetry is an important intermediate step in generating view-invariant representations. Finally, we measured neural responses to unnatural or non-canonical rotations of the face. View-symmetric patterns of response were also found in face-selective regions. However, in contrast to natural or canonical rotations of the face, these view-symmetric responses did not arise from an initial view-dependent representation in early visual areas. This suggests differences in the way that view-symmetrical representations emerge with canonical or non-canonical rotations. The similarity in the neural response to canonical views of familiar and unfamiliar faces in the core face network suggests that the neural correlates of familiarity emerge at later stages of processing.

Is understanding the mechanism of perception possible?

Perception is the understanding by the brain of the different sensory information. In humans, information is mostly visual and is conveyed to the occipital cortex. MRI techniques suggest that it is deciphered in different areas of the temporal cortex, depending upon the observed scene. The neural codes used by neurons of these areas will be discussed in two instances: Recognition of written words and of faces. In the first case, an hypothesis using basic properties of neurons and hierarchy of local combination detectors, accounts for the invariance of their visual form. That it is relevant is emphasized by its successful application to reading mechanism. In the second more recent instance, recognition of human faces by macaques is shown to reside in neuron patches in the inferior temporal cortex, forming a system identifying faces with view invariance. The code allowing facial identification using the electrical activity of 200 neurons was cracked.

Navigation and star identification for an interstellar mission

The possibility of interstellar scientific missions, even if only to explore the local Interstellar Medium (ISM), has received increased consideration in recent years. Due to the immense distances involved, an interstellar spacecraft must be capable of autonomous navigation. Many autonomous navigation strategies exist, but nearly all of these depend in some way on star observations and an ability to match these observations to a catalog of known stars. This is generally accomplished by computing and matching a star pattern descriptor that is invariant to changes in the spacecraft attitude. Unlike missions within the Solar System, an interstellar spacecraft will require a catalog that treats stars as 3D points rather than directions. This is problematic, since it has long been known that view invariants do not exist for sets of 3D points—making the classical method (or any modifications thereof) completely ineffectual for star identification in interstellar space. Moreover, other effects for a spacecraft traveling at relativistic speeds – such as stellar aberration and the relativistic Doppler effect – can negatively affect star identification, navigation observability, and sensor design. These challenges of star identification during interstellar flight are discussed and some possible solutions are suggested. Numerical examples are provided for a variety of motivating mission concepts.

Multi-view motion modelled deep attention networks (M2DA-Net) for video based sign language recognition

Currently, video-based Sign language recognition (SLR) has been extensively studied using deep learning models such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In addition, using multi view attention mechanism along with CNNs could be an appealing solution that can be considered in order to make the machine interpretation process immune to finger self-occlusions. The proposed multi stream CNN mixes spatial and motion modelled video sequences to create a low dimensional feature vector at multiple stages in the CNN pipeline. Hence, we solve the view invariance problem into a video classification problem using attention model CNNs. For superior network performance during training, the signs are learned through a motion attention network thus focusing on the parts that play a major role in generating a view based paired pooling using a trainable view pair pooling network (VPPN). The VPPN, pairs views to produce a maximally distributed discriminating features from all the views for an improved sign recognition. The results showed an increase in recognition accuracies on 2D video sign language datasets. Similar results were obtained on benchmark action datasets such as NTU RGB D, MuHAVi, WEIZMANN and NUMA as there is no multi view sign language dataset except ours.

Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio

The rhesus macaque is an important model species in several branches of science, including neuroscience, psychology, ethology, and medicine. The utility of the macaque model would be greatly enhanced by the ability to precisely measure behavior in freely moving conditions. Existing approaches do not provide sufficient tracking. Here, we describe OpenMonkeyStudio, a deep learning-based markerless motion capture system for estimating 3D pose in freely moving macaques in large unconstrained environments. Our system makes use of 62 machine vision cameras that encircle an open 2.45 m × 2.45 m × 2.75 m enclosure. The resulting multiview image streams allow for data augmentation via 3D-reconstruction of annotated images to train a robust view-invariant deep neural network. This view invariance represents an important advance over previous markerless 2D tracking approaches, and allows fully automatic pose inference on unconstrained natural motion. We show that OpenMonkeyStudio can be used to accurately recognize actions and track social interactions.

A map of object space in primate inferotemporal cortex.

The inferotemporal (IT) cortex is responsible for object recognition, but it is unclear how the representation of visual objects is organized in this part of the brain. Areas that are selective for categories such as faces, bodies, and scenes have been found1-5, but large parts of IT cortex lack any known specialization, raising the question of what general principle governs IT organization. Here we used functional MRI, microstimulation, electrophysiology, and deep networks to investigate the organization of macaque IT cortex. We built a low-dimensional object space to describe general objects using a feedforward deep neural network trained on object classification6. Responses of IT cells to a large set of objects revealed that single IT cells project incoming objects onto specific axes of this space. Anatomically, cells were clustered into four networks according to the first two components of their preferred axes, forming a map of object space. This map was repeated across three hierarchical stages of increasing view invariance, and cells that comprised these maps collectively harboured sufficient coding capacity to approximately reconstruct objects. These results provide a unified picture of IT organization in which category-selective regions are part of a coarse map of object space whose dimensions can be extracted from a deep network.

Multi-view region-adaptive multi-temporal DMM and RGB action recognition

Human action recognition remains an important yet challenging task. This work proposes a novel action recognition system. It uses a novel multi-view region-adaptive multi-resolution-in-time depth motion map (MV-RAMDMM) formulation combined with appearance information. Multi-stream 3D convolutional neural networks (CNNs) are trained on the different views and time resolutions of the region-adaptive depth motion maps. Multiple views are synthesised to enhance the view invariance. The region-adaptive weights, based on localised motion, accentuate and differentiate parts of actions possessing faster motion. Dedicated 3D CNN streams for multi-time resolution appearance information are also included. These help to identify and differentiate between small object interactions. A pre-trained 3D-CNN is used here with fine-tuning for each stream along with multi-class support vector machines. Average score fusion is used on the output. The developed approach is capable of recognising both human action and human–object interaction. Three public-domain data-sets, namely MSR 3D Action, Northwestern UCLA multi-view actions and MSR 3D daily activity, are used to evaluate the proposed solution. The experimental results demonstrate the robustness of this approach compared with state-of-the-art algorithms.

An intrinsic algorithm for viewing angle tolerance of object discrimination in human subjects.

Prevailing theories suggest that view-invariant object recognition is accomplished via spatiotemporal correlations of multiple views that bind different views to the same object. However, it is unknown how the exposure to multiple views without association affects view-invariant recognition generating. Behavioural studies have shown that monkeys acquired view-invariant object recognition capability in a range of the viewing angles of 30° to 60° after experiencing discrimination of similar objects at each of several viewing angles without associating different views, but the monkeys could not discriminate novel objects from similar distractors when the viewing angle changed. In the present study the development of the view invariance was tested in the human subjects and compared with the results of the monkeys previously reported. The view-invariant object recognition capability of human subjects was tested using either familiar objects that the subjects experienced in a preparatory object discrimination task at the same viewpoints or novel objects that the subjects had never experienced. In the viewing angle range of 30°, human subjects showed significant object discrimination capability across views, with no need for prior experience of the objects. Prior discrimination experience within the same viewpoints endowed the human subjects with broadening of the viewing angle tuning, because an object discrimination test immediately after the prior discrimination experience at a single view showed wider viewing angle tuning than a test without prior experience. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Few-Shot Deep Adversarial Learning for Video-based Person Re-identification.

Video-based person re-identification (re-ID) refers to matching people across camera views from arbitrary unaligned video footages. Existing methods rely on supervision signals to optimise a projected space under which the distances between inter/intra-videos are maximised/minimised. However, this demands exhaustively labelling people across camera views, rendering them unable to be scaled in large networked cameras. Also, it is noticed that learning effective video representations with view invariance is not explicitly addressed for which features exhibit different distributions otherwise. Thus, matching videos for person re-ID demands flexible models to capture the dynamics in time-series observations and learn view-invariant representations with access to limited labeled training samples. In this paper, we propose a novel few-shot deep learning approach to videobased person re-ID, to learn comparable representations that are discriminative and view-invariant. The proposed method is developed on the variational recurrent neural networks (VRNNs) and trained adversarially to produce latent variables with temporal dependencies that are highly discriminative yet view-invariant in matching persons. Through extensive experiments conducted on three benchmark datasets, we empirically show the capability of our method in creating view-invariant temporal features and state-of-the-art performance achieved by our method.

Learning to detect anatomical landmarks of the pelvis in X-rays from arbitrary views.

Minimally invasive alternatives are now available for many complex surgeries. These approaches are enabled by the increasing availability of intra-operative image guidance. Yet, fluoroscopic X-rays suffer from projective transformation and thus cannot provide direct views onto anatomy. Surgeons could highly benefit from additional information, such as the anatomical landmark locations in the projections, to support intra-operative decision making. However, detecting landmarks is challenging since the viewing direction changes substantially between views leading to varying appearance of the same landmark. Therefore, and to the best of our knowledge, view-independent anatomical landmark detection has not been investigated yet. In this work, we propose a novel approach to detect multiple anatomical landmarks in X-ray images from arbitrary viewing directions. To this end, a sequential prediction framework based on convolutional neural networks is employed to simultaneously regress all landmark locations. For training, synthetic X-rays are generated with a physically accurate forward model that allows direct application of the trained model to real X-ray images of the pelvis. View invariance is achieved via data augmentation by sampling viewing angles on a spherical segment of [Formula: see text]. On synthetic data, a mean prediction error of 5.6 ± 4.5mm is achieved. Further, we demonstrate that the trained model can be directly applied to real X-rays and show that these detections define correspondences to a respective CT volume, which allows for analytic estimation of the 11 degree of freedom projective mapping. We present the first tool to detect anatomical landmarks in X-ray images independent of their viewing direction. Access to this information during surgery may benefit decision making and constitutes a first step toward global initialization of 2D/3D registration without the need of calibration. As such, the proposed concept has a strong prospect to facilitate and enhance applications and methods in the realm of image-guided surgery.

Multi-view recognition system for human activity based on multiple features for video surveillance system

Recognition of the activities of human, image sequences is most active area of research in computer vision and most of the previous projects, the focus of the activity is to acknowledge the recognition, from a single view and ignored issues of multiple view invariance. In this paper, the proposed framework for the recognition of a view-invariant human activity, solves the above problem. The components of the proposed framework are three consecutive modules: (i) the detection and positioning of the person’s background subtraction, (ii) the function extraction (iii) and the final activity is referenced by using a set of hidden Markov models (HMMs). During features extraction phase in the proposed method for activity representation a combination of contour-based distance signal feature, optical flow-based motion feature and uniform rotation local binary patterns has been used. Due to its rotation invariant nature the uniform LBP provides view-invariant recognition of multi-view human activities. A successful testing of the proposed approach was done on our own viewpoint dataset, KTH action recognition dataset, i3DPost multi-view dataset, and MSR view-point action dataset. From the experimental results and analysis over the chosen datasets, it is observed that the proposed framework is robust, flexible and efficient with respect to multiple views activity recognition, scale and phase variations.

Object recognition in fish: accurate discrimination across novel views of an unfamiliar object category (human faces)

Accurate visual object recognition is essential to survival for a wide range of species across a wide range of evolutionary histories and visual requirements. However this task is solved, it is a major achievement because object recognition is far from simple. The appearance of an object can alter almost completely as viewing conditions change, not least under variations in lighting and orientation. Determining the recognition limits of a species is important to understanding its visual ecology and can help identify conditions under which recognition may fail. In this study, we tested whether a species of fish can recognize objects from an unfamiliar object class (human faces) across changes in viewing direction. Using operant conditioning, we trained archerfish, Toxotes chatareus, to discriminate between two frontal views of standardized human faces and, critically, tested whether they could continue to do so as the orientation in depth of the faces changed. All fish learned the initial discrimination task and could also recognize rotated forms. These results represent the first conclusive evidence that a species of fish can generalize recognition across views, speaking against a strict image-matching process. This ability rather speaks to the capacity of relatively simple brains to tackle the hard problem of view invariance and provides insight into the mechanisms employed in more complex organisms such as humans. Although we speculate that other fish species may demonstrate similar abilities, a visual system capable of recognition across changes in viewpoint may be especially important to the unique hunting strategy of archerfish.

View-invariant representation of hand postures in the human lateral occipitotemporal cortex

Understanding other people's actions and mental states includes the interpretation of body postures and movements. In particular, hand postures are an important channel to signal both action and communicative intentions. Recognizing hand postures is computationally challenging because hand postures often differ only in the subtle configuration of relative finger positions and because visual characteristics of hand postures change across viewpoints. To allow for accurate interpretation, the brain needs to represent hand postures in a view-invariant but posture-specific manner. Here we test for such representations in hand-, body-, and object-selective regions of the lateral occipitotemporal cortex (LOTC). We used multivariate pattern analysis of fMRI data to test for view-specific and view-invariant representations of individual hand postures, separately for two domains: action-related postures (e.g., a precision grasp) and communicative postures (e.g., thumbs up). Results showed that hand-selective LOTC, but not nearby body- and object-selective LOTC, represented hand postures in a view-invariant manner, with relatively similar activity patterns to the same hand posture seen from different viewpoints. View invariance was equally strong for action and communicative postures. By contrast, object-selective cortex represented hand postures in a view-specific manner. These results indicate a role for hand-selective LOTC in solving the view-invariance problem for individual hand postures. View-invariant representations of hand postures in this region may then be accessed and further interpreted by multiple downstream systems to inform high-level judgments related to action understanding, emotion recognition, and non-verbal communication.

Cross-View Action Recognition Based on Hierarchical View-Shared Dictionary Learning

Recognizing human actions across different views is challenging, since observations of the same action often vary greatly with viewpoints. To solve this problem, most existing methods explore the cross-view feature transfer relationship at video level only, ignoring the sequential composition of action segments therein. In this paper, we propose a novel hierarchical transfer framework, which is based on an action temporal-structure model that contains sequential relationship between action segments at multiple timescales. Thus, it can capture the view invariance of the sequential relationship of segment-level transfer. Additionally, we observe that the original feature distributions under different views differ greatly, leading to view-dependent representations irrelevant to the intrinsic structure of actions. Thus, at each level of the proposed framework, we transform the original feature spaces of different views to a view-shared low-dimensional feature space, and jointly learn a dictionary in this space for these views. This view-shared dictionary captures the common structure of action data across the views and can represent the action segments in a way robust to view changes. Moreover, the proposed method can be kernelized easily, and operate in both unsupervised and supervised cross-view scenarios. Extensive experimental results on the IXMAS and WVU datasets demonstrate superiority of the proposed method over state-of-the-art methods.

High-Level Prediction Signals in a Low-Level Area of the Macaque Face-Processing Hierarchy

Theories like predictive coding propose that lower-order brain areas compare their inputs to predictions derived from higher-order representations and signal their deviation as a prediction error. Here, we investigate whether the macaque face-processing system, a three-level hierarchy in the ventral stream, employs such a coding strategy. We show that after statistical learning of specific face sequences, the lower-level face area ML computes the deviation of actual from predicted stimuli. But these signals do not reflect the tuning characteristic of ML. Rather, they exhibit identity specificity and view invariance, the tuning properties of higher-level face areas AL and AM. Thus, learning appears to endow lower-level areas with the capability to test predictions at a higher level of abstraction than what is afforded by the feedforward sweep. These results provide evidence for computational architectures like predictive coding and suggest a new quality of functional organization of information-processing hierarchies beyond pure feedforward schemes.

Building of object view invariance in a newly-discovered network in inferior temporal cortex

Building of object view invariance in a newly-discovered network in inferior temporal cortex

Wireless Kinect-NAO Framework Based on Takagi-Sugeno Fuzzy Inference System

Abstract In the context of learning new skills by imitation for children with special educational needs, we propose Wireless Kinect-NAO Framework (WKNF) for robot teleoperation in real time based on Takagi-Sugeno (T-S) Fuzzy Inference System. The new solutions here are related to complex whole-body motion retargeting, standing body stabilization, view invariance and smoothness of robot motions. The raw depth Kinect data are fuzzified and processed by median filter. The joint angles estimation for motion mapping of Human to NAO movements is based on fuzzy logic and featured angles rather than direct angles are calculated by Inverse Kinematics due to differences in the human and robot kinematics. During the joint angles calculation nonlinearities are observed as a result of ambiguity of Kinect 3D joint coordinates in different offsets. NAO kinematic limitations and nonlinearities in workspace are decomposed and linearly approximated by T-S fuzzy rules of zero and first order that have local support in 2D projections. To prevent the robot to fall down, the center of mass is considered in order NAO to stay within a support and safe polygon. The feasibility of the proposed framework has been proven by real experiments.

Evidence for view-invariant face recognition units in unfamiliar face learning.

Many models of face recognition incorporate the idea of a face recognition unit (FRU), an abstracted representation formed from each experience of a face which aids recognition under novel viewing conditions. Some previous studies have failed to find evidence of this FRU representation. Here, we report three experiments which investigated this theoretical construct by modifying the face learning procedure from that in previous work. During learning, one or two views of previously unfamiliar faces were shown to participants in a serial matching task. Later, participants attempted to recognize both seen and novel views of the learned faces (recognition phase). Experiment 1 tested participants' recognition of a novel view, a day after learning. Experiment 2 was identical, but tested participants on the same day as learning. Experiment 3 repeated Experiment 1, but tested participants on a novel view that was outside the rotation of those views learned. Results revealed a significant advantage, across all experiments, for recognizing a novel view when two views had been learned compared to single view learning. The observed view invariance supports the notion that an FRU representation is established during multi-view face learning under particular learning conditions.

Effect of silhouetting and inversion on view invariance in the monkey inferotemporal cortex.

We effortlessly recognize objects across changes in viewpoint, but we know relatively little about the features that underlie viewpoint invariance in the brain. Here, we set out to characterize how viewpoint invariance in monkey inferior temporal (IT) neurons is influenced by two image manipulations-silhouetting and inversion. Reducing an object into its silhouette removes internal detail, so this would reveal how much viewpoint invariance depends on the external contours. Inverting an object retains but rearranges features, so this would reveal how much viewpoint invariance depends on the arrangement and orientation of features. Our main findings are 1) view invariance is weakened by silhouetting but not by inversion; 2) view invariance was stronger in neurons that generalized across silhouetting and inversion; 3) neuronal responses to natural objects matched early with that of silhouettes and only later to that of inverted objects, indicative of coarse-to-fine processing; and 4) the impact of silhouetting and inversion depended on object structure. Taken together, our results elucidate the underlying features and dynamics of view-invariant object representations in the brain.NEW & NOTEWORTHY We easily recognize objects across changes in viewpoint, but the underlying features are unknown. Here, we show that view invariance in the monkey inferotemporal cortex is driven mainly by external object contours and is not specialized for object orientation. We also find that the responses to natural objects match with that of their silhouettes early in the response, and with inverted versions later in the response-indicative of a coarse-to-fine processing sequence in the brain.