Dynamic Visual Scenes Research Articles

Processing moving visual scenes during upright stance in elderly patients with mild cognitive impairment.

BackgroundThe ability to maintain balance in an upright stance gradually worsens with age and is even more difficult for patients with cognitive disorders. Cognitive impairment plays a probable role in the worsening of stability. The purpose of this study was to expose subjects with mild cognitive impairment (MCI) and healthy, age-matched controls to moving visual scenes in order to examine their postural adaptation abilities.MethodsWe observed postural responses to moving visual stimulation while subjects stood on a force platform. The visual disturbance was created by interposing a moving picture in four directions (forward, backward, right, and left). The pre-stimulus (a static scene for 10 s), stimulus (a dynamic visual scene for 20 seconds) and post-stimulus (a static scene for 20 seconds) periods were evaluated. We separately analyzed the total path (TP) of the center of pressure (COP) and the root mean square (RMS) of the COP displacement in all four directions.ResultsWe found differences in the TP of the COP during the post-stimulus period for all stimulus directions except in motion towards the subject (left p = 0.006, right p = 0.004, and away from the subject p = 0.009). Significant RMS differences between groups were also observed during the post-stimulus period in all directions except when directed towards the subject (left p = 0.002, right p = 0.007, and away from the subject p = 0.014).ConclusionExposing subjects to a moving visual scene induced greater destabilization in MCI subjects compared to healthy elderly controls. Surprisingly, the moving visual scene also induced significant aftereffects in the MCI group. Our findings indicate that the MCI group had diminished adaptation to the dynamic visual scene and recovery. These results suggest that even mild cognitive deficits can impair sensory information integration and alter the sensory re-weighing process.

Affective question answering on video

Visual Question Answering (VQA) is an increasingly popular research area in machine learning. Most of the existing VQA tasks only focus on static images, and only a few models are based on videos. The primary purpose of this project is to develop an innovative model that performs Affective Question Answering on Video (AQAV), a multi-tasking architecture that implements a Video QA route and an Affective route. A pre-trained CNN emotion detector recognizes emotions on the frames of a video, and a string of the emotion labels is relayed to the Token-based, Frame-based and Integrated attention mechanisms. The attention model uses the visual features, the question and the emotion labels to focus on relevant frames of the video and relevant regions of the frames. The string of emotion labels is used to generate an emotion caption that will be used by the Text QA module to prepare an affective answer. A conventional answer is generated from processes that take place along the Video QA route, while the affective answer is a product of both the Video QA and the Affective routes. Our model does not only make VQA more analytic by generating an explanatory answer, but also registers quantitative improvement in performance, when compared with previous baselines. We managed to prove that the injection of emotions in the attention mechanism boosts VQA performance. The AQAV model contributes towards efforts in making machines understand sequential and dynamic visual scenes in the real world.

Multifeature fusion for robust human detection in thermal infrared imagery

Despite all the significant advances in human detection in various environmental conditions, it is still a challenging task. Most of the human detection algorithms mainly use color information, which is not robust to lighting changes and varying colors under which such a detector should operate namely day and nighttime. This problem is further amplified with infrared (IR) imagery, which only contains grayscale information. The proposed algorithm for human detection uses intensity distribution, gradient, and texture features for effective detection of humans in IR imagery. For the detection of intensity, histogram information is obtained in the grayscale channel. For extracting gradients, we utilize the histogram of oriented gradients for better information in the various lighting scenarios. For extracting texture information, center-symmetric local binary pattern gives rotational invariance as well as lighting invariance for robust features under these conditions. Various binning strategies help keep the inherent structure embedded in the features, which provide enough information for robust detection of the humans in the scene. The features are then classified using an AdaBoost classifier to provide a tree-like structure for detection in multiple scales. The algorithm has been trained and tested on IR imagery and has been found to be fairly robust to viewpoint changes and lighting changes in dynamic backgrounds and visual scenes.

Coarse to fine human face detection in a dynamic visual scene

Coarse to fine human face detection in a dynamic visual scene

Category-selective human brain processes elicited in fast periodic visual stimulation streams are immune to temporal predictability.

Recording direct neural activity when periodically inserting exemplars of a particular category in a rapid visual stream of other objects offers an objective and efficient way to quantify perceptual categorization and characterize its spatiotemporal dynamics. However, since periodicity entails predictability, perceptual categorization processes identified within this framework may be partly generated or modulated by temporal expectations. Here we present a stringent test of the hypothesis that temporal predictability generates or modulates category-selective neural processes as measured in a rapid periodic visual stimulation stream. In Experiment 1, we compare neurophysiological responses to periodic and nonperiodic (i.e., unpredictable) variable face stimuli in a fast (12Hz) visual stream of nonface objects. In Experiment 2, we assess potential responses to rare (10%) omissions of periodic face events (i.e., violations of periodicity) in the same fast visual stream. Overall, our observations indicate that category(face)-selective processes elicited in a fast periodic stream of visual objects are immune to temporal predictability. These observations do not support a predictive coding framework interpretation of category-change detection in the human brain and have important implications for understanding automatic human perceptual categorization in a rapidly changing (i.e., dynamic) visual scene.

Influence of speed-related auditory feedback on braking in a 3D-driving simulator

Although discrete auditory stimuli have been found useful for emergency braking, the role of continuous speed-related auditory feedback has not been investigated yet. This point may though be of importance in electric vehicles in which acoustic cues are drastically changed. The present study addressed this question through two experiments. In experiment 1, 12 usual drivers were exposed to naturalistic auditory feedback mimicking those issued from electric cars, while facing dynamic visual scenes in a 3D driving simulator. After being passively travelled up to a sustained constant speed, subjects had to stop their car in front of a traffic light that unexpectedly turned to red. Modifications of the speed-related auditory feedback did not impact braking initiation and regulation. In experiment 2, synthesized auditory feedback based on the Shepard-Risset glissando was provided to a new sample of 15 usual drivers in the same task. Pitch variations of this acoustic stimulus, although not scaled to an absolute speed, were manipulated as a function of visual speed changes. Changing the mapping between pitch variations of the synthesized auditory feedback and visual speed changes induced adjustments on braking which depended on acceleration/deceleration feedback. These findings stressed the importance of the acoustic content and its dynamics for car speed control.

Object individuation is invariant to attentional diffusion: Changes in the size of the attended region do not interact with object-substitution masking

When the human brain is confronted with complex and dynamic visual scenes, two pivotal processes are at play: visual attention (the process of selecting certain aspects of the scene for privileged processing) and object individuation (determining what information belongs to a continuing object over time versus what represents two or more distinct objects). Here we examined whether these processes are independent or whether they interact. Object-substitution masking (OSM) has been used as a tool to examine such questions, however, there is controversy surrounding whether OSM reflects object individuation versus substitution processes. The object-individuation account is agnostic regarding the role of attention, whereas object-substitution theory stipulates a pivotal role for attention. There have been attempts to investigate the role of attention in OSM, but they have been subject to alternative explanations. Here, therefore, we manipulated the size of the attended region, a pure and uncontaminated attentional manipulation, and examined the impact on OSM. Across three experiments, there was no interaction. This refutes the object-substitution theory of OSM. This, in turn, tell us that object-individuation is invariant the distribution of attention.

Multistable Perception in Older Adults: Constructing a Whole from Fragments.

Visual perception is constructive in nature; that is, a coherent whole is generated from ambiguous fragments that are encountered in dynamic visual scenes. Creating this coherent whole from fragmented sensory inputs requires one to detect, identify, distinguish and organize sensory input. The organization of fragments into a coherent whole is facilitated by the continuous interactions between lower level sensory inputs and higher order processes. However, age-related declines are found in both neural structures and cognitive processes (e.g., attention and inhibition). The impact of these declines on the constructive nature of visual processing was the focus of this study. Here we asked younger adults, young-old (65–79 years), and old-old adults (80+ years) to view a multistable figure (i.e., Necker cube) under four conditions (free, priming, volition, and adaptation) and report, via a button press, when percepts spontaneously changed. The oldest-olds, unlike young-olds and younger adults, were influenced by priming, had less visual stability during volition and showed less ability to adapt to multistable stimuli. These results suggest that the ability to construct a coherent whole from fragments declines with age. More specifically, vision is constructed differently in the old-olds, which might influence environmental interpretations and navigational abilities in this age group.

3D-Multiple Object Tracking training task improves passing decision-making accuracy in soccer players

ObjectivesThe ability to perform a context-free 3-dimensional multiple object tracking (3D-MOT) task has been highly related to athletic performance. In the present study, we assessed the transferability of a perceptual-cognitive 3D-MOT training from a laboratory setting to a soccer field, a sport in which the capacity to correctly read the dynamic visual scene is a prerequisite to performance. DesignThroughout pre- and post-training sessions, we looked at three essential skills (passing, dribbling, shooting) that are used to gain the upper hand over the opponent. MethodWe recorded decision-making accuracy during small-sided games in university-level soccer players (n = 23) before and after a training protocol. Experimental (n = 9) and active control (n = 7) groups were respectively trained during 10 sessions of 3D-MOT or 3D soccer videos. A passive control group (n = 7) did not received any particular training or instructions. ResultsDecision-making accuracy in passing, but not in dribbling and shooting, between pre- and post-sessions was superior for the 3D-MOT trained group compared to control groups. This result was correlated with the players' subjective decision-making accuracy, rated after pre- and post-sessions through a visual analogue scale questionnaire. ConclusionsTo our knowledge, this study represents the first evidence in which a non-contextual, perceptual-cognitive training exercise has a transfer effect onto the field in athletes.

The N170 observed 'in the wild': robust event-related potentials to faces in cluttered dynamic visual scenes.

As a social species in a constantly changing environment, humans rely heavily on the informational richness and communicative capacity of the face. Thus, understanding how the brain processes information about faces in real-time is of paramount importance. The N170 is a high-temporal resolution electrophysiological index of the brain's early response to visual stimuli that is reliably elicited in carefully controlled laboratory-based studies. Although the N170 has often been reported to be of greatest amplitude to faces, there has been debate regarding whether this effect might be an artefact of certain aspects of the controlled experimental stimulation schedules and materials. To investigate whether the N170 can be identified in more realistic conditions with highly variable and cluttered visual images and accompanying auditory stimuli we recorded EEG 'in the wild', while participants watched pop videos. Scene-cuts to faces generated a clear N170 response, and this was larger than the N170 to transitions where the videos cut to non-face stimuli. Within participants, wild-type face N170 amplitudes were moderately correlated to those observed in a typical laboratory experiment. Thus, we demonstrate that the face N170 is a robust and ecologically valid phenomenon and not an artefact arising as an unintended consequence of some property of the more typical laboratory paradigm.

Gaze-to-Object Mapping during Visual Search in 3D Virtual Environments

Stimuli obtained from highly dynamic 3D virtual environments and synchronous eye-tracking data are commonly used by algorithms that strive to correlate gaze to scene objects, a process referred to as gaze-to-object mapping (GTOM) . We propose to address this problem with a probabilistic approach using Bayesian inference. The desired result of the inference is a predicted probability density function (PDF) specifying for each object in the scene a probability to be attended by the user. To evaluate the quality of a predicted attention PDF, we present a methodology to assess the information value (i.e., likelihood) in the predictions of different approaches that can be used to infer object attention. To this end, we propose an experiment based on a visual search task, which allows us to determine the object of attention at a certain point in time under controlled conditions. We perform this experiment with a wide range of static and dynamic visual scenes to obtain a ground-truth evaluation dataset, allowing us to assess GTOM techniques in a set of 30 particularly challenging cases.

Towards an understanding of the roles of visual areas MT and MST in computing speed.

Towards an understanding of the roles of visual areas MT and MST in computing speed.

A human-computer collaborative workflow for the acquisition and analysis of terrestrial insect movement in behavioral field studies

The study of insect behavior from video sequences poses many challenges. Despite the advances in image processing techniques, the current generation of insect tracking tools is only effective in controlled lab environments and under ideal lighting conditions. Very few tools are capable of tracking insects in outdoor environments where the insects normally operate. Furthermore, the majority of tools focus on the first stage of the analysis workflow, namely the acquisition of movement trajectories from video sequences. Far less effort has gone into developing specialized techniques to characterize insect movement patterns once acquired from videos. In this paper, we present a human-computer collaborative workflow for the acquisition and analysis of insect behavior from field-recorded videos. We employ a human-guided video processing method to identify and track insects from noisy videos with dynamic lighting conditions and unpredictable visual scenes, improving tracking precision by 20% to 44% compared to traditional automated methods. The workflow also incorporates a novel visualization tool for the large-scale exploratory analysis of insect trajectories. We also provide a number of quantitative methods for statistical hypothesis testing. Together, the various components of the workflow provide end-to-end quantitative and qualitative methods for the study of insect behavior from field-recorded videos. We demonstrate the effectiveness of the proposed workflow with a field study on the navigational strategies of Kenyan seed harvester ants.

Stereoscopy benefits processing of dynamic visual scenes by disambiguating object occlusions

How and to what extent stereoscopic cues help process dynamic visual scenes is still unknown. As we navigate in crowds or when we play sports we are often obliged to make rapid decisions in complex motion environments while attending to multiple elements. Stereoscopy can improve speed thresholds for a multiple object tracking (MOT) (Faubert & Sidebottom, 2012), but the nature of this advantage remains undetermined. From an attention perspective, there are at least two possible hypotheses to explain this advantage. One possibility is that stereoscopy reduces the attentional bottleneck as seen in 2-dimensional MOT environments (Intrilligator & Canavagh, 2001) by distributing attention uniformly in 3D space. Another hypothesis is that the stereoscopy helps attentional tracking by segregating the target and the non-target objects during occlusions. We have addressed these hypotheses by testing subjects with and without stereoscopic cues for different stimulus configurations in which pairs of spheres were rotating in orbit with one another in a 3D virtual space. In one set of conditions, object pairs rotated without ever occluding each other. In the second set of conditions, the objects occasionally occluded each other as they were rotating around an axis perpendicular to the observer. All conditions consisted of 4 pairs of spheres presented in each quadrant of the visual field at 20 degrees of eccentricity. Stereoscopy improved MOT speed threshold by a factor of about 3 when objects occluded each other, but slightly, yet significantly, impaired speed threshold by about 18% when objects did not occlude each other. We conclude that the overall benefit of stereoscopy for processing dynamic scenes comes from improved attention tracking by disambiguating targets from non-targets during occlusions. Conversely, stereoscopic cues were a disadvantage in absence of occlusions. Meeting abstract presented at VSS 2013

Stereoscopy benefits processing of dynamic visual scenes by disambiguating object occlusions

Stereoscopy benefits processing of dynamic visual scenes by disambiguating object occlusions

A System-Prototype Representing 3D Space via Alternative-Sensing for Visually Impaired Navigation

Offering an alternative mode of interaction with the surrounding 3-D space to the visually impaired for collision free navigation is a goal of great significance that includes several key challenges. In this paper, we study the alternative 3-D space sensation that is interpreted by our computer vision prototype system and transferred to the user via a vibration array. There are two main tasks for conducting such a study. The first task is to detect obstacles in close proximity, and motion patterns in image sequences, both important issues for a safe navigation in a 3-D dynamic space. To achieve this task, the images from the left and right cameras are acquired to produce new stereo images, followed by video stabilization as a preprocessing stage, a nonlinear spatio-temporal diffusion and kernel based density estimation method to assess the motion activity, and finally watershed-based detection of moving regions (or obstacles) of interest. The second task is to efficiently represent the information of the captured static and dynamic visual scenes as 3-D detectable patterns of vibrations applied on the human body to create a 3-D sensation of the space during navigation. To accomplish this task, considering the current limitations imposed by the technology, we create a high-to-low (H-L) image resolution representation scheme to facilitate the mapping onto a low-resolution 2-D array of vibrators. The H-L scheme uses pyramidal modeling to obtain low-resolution images of interest-preserving motion and obstacles-that are mapped onto a vibration array. These patterns are utilized to train and test the performance of the users in free space navigation. Thus, in this paper we study the synergy of these two important schemes to offer an alternative sensation of the 3-D space to the visually impaired via an array of vibrators. Particularly, the motion component is employed as an element for the identification of visual information of interest to be retained during the H-L transformation. The role of the array vibrators is to create a small-scale front representation of the space via various levels of vibrations. Thus, 3-D vibrations applied on the user's body (chest, abdomen) offer a 3-D sensation of the surrounding space and the motion in it. In addition, we present experimental results that indicate the efficiency of this navigation scheme in creating low-resolution 3-D views of the free navigation space and detecting obstacles and moving areas.

Professional athletes have extraordinary skills for rapidly learning complex and neutral dynamic visual scenes

Evidence suggests that an athlete's sports-related perceptual-cognitive expertise is a crucial element of top-level competitive sports1. When directly assessing whether such experience-related abilities correspond to fundamental and non-specific cognitive laboratory measures such as processing speed and attention, studies have shown moderate effects leading to the conclusion that their special abilities are context-specific2. We trained 308 observers on a complex dynamic visual scene task void of context and motor control requirements3 and demonstrate that professionals as a group dramatically differ from high-level amateur athletes, who dramatically differ from non-athlete university students in their capacity to learn such stimuli. This demonstrates that a distinguishing factor explaining the capacities of professional athletes is their ability to learn how to process complex dynamic visual scenes. This gives us an insight as to what is so special about the elite athletes' mental abilities, which allows them to express great prowess in action.

Compréhension des interactions sociales chez des enfants atteints de troubles du spectre de l’autisme : le langage du corps leur « parle »-t-il ?

Autism spectrum disorders (ASD) are characterized by difficulties in social interaction and verbal and non verbal reciprocal communication. Face and gaze direction, which participate in non verbal communication, are described as atypical in ASD. Also body movements carry multiple social cues. Under certain circumstances, for instance when seeing two persons from far, they constitute the only support that allows the grasping of a social content. Here, we investigated the contribution of whole-body motion processing in social understanding. The aim of the study was to evaluate whether children with ASD make use of information carried by body motion to categorize dynamic visual scenes that portrayed social interactions. In 1973, Johansson devised a technique for studying the perception of biological motion that minimizes static form information from the stimulus, but retains motion information. In these point-light displays, the movement figure, such as a body, is represented by a small number of illuminated dots positioned to highlight the motion of the body parts. We used Johansson's model to explore the ability of children with ASD to understand social interactions based on human movement analysis. Three-second silent point-light displays were created by videotaping two actors. The two actors were either interacting together or moving side by side without interacting. A large range of social interaction displays were used to cover social scenes depicting social norms (conventional gestures and courteous attitudes), emotional situations (carrying positive or negative valences) and scenes from games (sports, dance, etc.). Children were asked to carefully watch the stimuli and to classify them according to the question "Are the two persons communicating or not?". Four sessions of 3 minutes were performed by each child. Children with ASD were compared with typically developing control children matched with either non verbal mental age or chronological age. Response and reaction time were recorded in this force-choice categorization task. The performance of children with ASD suggested that they were able to extract a social content from body motion. However, they were significantly less efficient than typically developing control children, either matched for non verbal mental age or chronological age. This was especially the case for the social interaction displays. Neither impaired global perceptual processing, nor cognitive development, nor emotional content could explain these lower performances. The results are discussed in the context of an action representation deficit and a dysfunction of the mirror mechanism in ASD. In conclusion, this behavioural study highlights the potential of point-light displays as a rehabilitation tool in ASD.

Gazing at tangent point location during curve driving does not avoid foveal motion and leads to optokinetic nystagmus

Many studies have shown that during curve driving, the drivers have gaze fixation patterns in the vicinity of the tangent point (TP), this latter being the intersection between the inside edge of the road and its tangent passing through the driver position. The interest for the TP is mainly due to the fact that on one hand, its angular position is linked to the road curvature and therefore can be a control variable for the driver; and, on the other hand, it may correspond to a minimal speed in the optical flow. However, the TP is only motionless when the trajectory precisely follows the curve's geometry. In the present study, we measured gaze behavior during curve driving, with the general hypothesis that gaze cannot be stable when exposed to a global optical flow due to self-motion. We used a driving simulator coupled to a gaze recording system. Ten participants drove on a track composed of eight curves of various radii. Results show that gaze position is, as previously described, located near the TP. In addition, we observe the presence of a systematic optokinetic nystagmus (OKN) around the TP position. The OKN slow-phase direction does not match the foveal optic flow direction, while slow-phase speed is about half the local speed. When averaging the flow on larger areas, gaze and flow directions match better, and gaze and flow speeds are optimally matched for an optic flow integration over two degrees. We thus confirm that the TP is a privileged feature in the dynamic visual scene during curve driving. However, studying only gaze fixation is not fully relevant, because the TP is surrounded by optic flow, whose global characteristics induce retinal drifts and lead to OKN behavior. We consider that this must be taken into account for future models of vehicle control.

Optokinetic nystagmus is elicited by curvilinear optic flow during high speed curve driving

When analyzing gaze behavior during curve driving, it is commonly accepted that gaze is mostly located in the vicinity of the tangent point, being the point where gaze direction tangents the curve inside edge. This approach neglects the fact that the tangent point is actually motionless only in the limit case when the trajectory precisely follows the curve’s geometry. In this study, we measured gaze behavior during curve driving, with the general hypothesis that gaze is not static, when exposed to a global optical flow due to self-motion. In order to study spatio-temporal aspects of gaze during curve driving, we used a driving simulator coupled to a gaze recording system. Ten participants drove seven runs on a track composed of eight curves of various radii (50, 100, 200 and 500 m), with each radius appearing in both right and left directions. Results showed that average gaze position was, as previously described, located in the vicinity of the tangent point. However, analysis also revealed the presence of a systematic optokinetic nystagmus (OKN) around the tangent point position. The OKN slow phase direction does not match the local optic flow direction, while slow phase speed is about half of the local speed. Higher directional gains are observed when averaging the entire optical flow projected on the simulation display, whereas the best speed gain is obtained for a 2° optic flow area, centered on the instantaneous gaze location. The present study confirms that the tangent point is a privileged feature in the dynamic visual scene during curve driving, and underlines a contribution of the global optical flow to gaze behavior during active self-motion.