Static Visual Features Research Articles

Tiny drone object detection in videos guided by the bio-inspired magnocellular computation model

Detecting drones in infrared videos is highly desired in many realistic scenarios, e.g., unauthorized drone monitoring around airports. Nevertheless, automated drone detection is rather challenging when the targets appear as tiny objects (≤10×10 pixels) against complex backgrounds. Conventional object detection algorithms, which mainly use static visual features, can hardly distinguish tiny objects from undesired artefacts in complex backgrounds. To alleviate this problem, we learn from the early biological visual pathway (including the parvocellular and magnocellular pathways), which process static and motion information simultaneously. Therefore, we propose a magnocellular inspired method for video tiny-object detection (Magno-VTOD) that integrates both static and motion visual information. The Magno-VTOD firstly employs a retinal magnocellular computation model to extract the motion strength of moving objects. The motion responses are then used to enhance the areas of the flying tiny drones effectively and efficiently, thereby facilitating the subsequent target detection procedure. We implement the video tiny-object detection method based on the widely adopted deep neural networks guided by the magnocellular computation model. Experimental results obtained on the large-scale Anti-UAV dataset (304451 video frames) validate that the proposed Magno-VTOD method significantly outperforms the competing state-of-the-art object detection methods on the tiny drone detection task. Particularly, the AP value is increased by 15.4% for tiny object detection, and by 17.1%/13.7% against wood/mountain backgrounds.

Depression Detection with Dynamic and Static Visual Features

Depression is a serious mood disorder that can significantly impact a person’s ability to live a normal life. In severe cases, it can even lead to suicidal thoughts. As such, accurate detection of depression is crucial for effective management and treatment. This paper presents a facial expression-based approach for depression detection, which is composed of two steps. First, static features are extracted using Local Binary Pattern (LBP), Histogram of Oriented Gradients (HOG), and Bag of Words (BOW). Second, dynamic features are obtained by applying LBPs on Three Orthogonal Planes (LBP-TOP) and Eight Vertices LBP (EVLBP) frame by frame. Next, the static and dynamic features are combined to create a 1377-dimensional vector for each video. Finally, Gradient Boosting Regression is used to predict depression scores. The experimental results on the AVEC 2014 depression dataset ([Formula: see text], [Formula: see text]) demonstrate the effectiveness of the proposed method. These results indicate that the low-dimensional vectors extracted by the proposed method can effectively capture the facial motion of individuals with depression, and also suggest that hand-crafted methods could have potential in depression detection.

Position-aware image captioning with spatial relation

Image caption aims to generate a language description of a given image. The problem can be solved by learning semantic information of visual objects and generating descriptions based on extracted embedding. However, the spatial relationship between visual objects and their static position is not fully explored by existing methods. In this work, we propose a Position-Aware Transformer (PAT) model that extracts both regional and static global visual features and unify both the regional and global by incorporating spatial information aligned to each visual feature. To make a better representation of spatial information and correlation between extracted visual features, we propose and compare three subtle approaches to explore position embedding with spatial relation information explicitly. Moreover, we jointly consider the static global and regional embedding for spatial modeling. Experimental results illustrate that our proposed model achieves competitive performance on the COCO image captioning dataset, where the PAT model could respectively reach 38.7, 28.6, and 58.6 on BLEU-4, METEOR, and ROUGE-L respectively. Extensive experiments suggest that the proposed PAT model could also reach competitive performance on related visual-language tasks including visual question answering (VQA) and multi-modal retrieval. Detailed ablation studies are conducted to report how each part would contribute to the final performance, which could be a good reference for follow-up spatial information representation works.

Recurrent Attention and Semantic Gate for Remote Sensing Image Captioning

The remote sensing image captioning has attracted wide spread attention in remote sensing field due to its application potentiality. However, most existing approaches model limited interactions between image content and sentence and fail to exploit special characteristics of the remote sensing images. We introduce a novel recurrent attention and semantic gate (RASG) framework to facilitate the remote sensing image captioning in this article, which integrates competitive visual features and a recurrent attention mechanism to generate a better context vector for the images every time as well as enhances the representations of the current word state. Specifically, we first project each image into competitive visual features by taking the advantage of both static visual features and multiscale features. Then, a novel recurrent attention mechanism is developed to extract the high-level attentive maps from encoded features and nonvisual features, which can help the decoder recognize and focus on the effective information for understanding the complex content of the remote sensing images. Finally, the hidden states from the long short-term memory (LSTM) and other semantic references are incorporated into a semantic gate, which contributes to more comprehensive and precise semantic understanding. Comprehensive experiments on three widely used datasets, Sydney-Captions, UCM-Captions, and Remote Sensing Image Captioning Dataset, have demonstrated the superiority of the proposed RASG over a series of attentive models based on image captioning methods.

A Visual-Inertial Localization Method for Unmanned Aerial Vehicle in Underground Tunnel Dynamic Environments

Unmanned Aerial Vehicles (UAVs) can significantly improve the autonomy of the mining industry, and self-localization is the key to autonomous flights of underground UAVs. A localization method of visual-inertial sensor data fusion is proposed in this paper. The method aims to improve the localization accuracy and robustness of underground UAVs in dynamic environments. First, an algorithm for dynamic point detection and rejection is presented, which combines a semantic segmentation neural network, an optical flow method, and an epipolar constraint method. Second, a visual-inertial sensor fusion algorithm is used to enhance performance in areas lacking static visual features. It can also provide absolute scales to the localization results, as opposed to monocular systems. Finally, a hand-held multi-sensor data collection system is developed with accurate calibration, for imitating flights of underground UAVs and easing data collection in real underground tunnels. We evaluate our proposed localization method and compare it with state-of-art method VINS-Mono on both the public EuRoC dataset and our own collected data in underground tunnels. Experimental results show that the proposed visual-inertial localization method can improve the accuracy by more than 67% over VINS-Mono in high dynamic environments, and it can be applied to underground dynamic scenes with high robustness and accuracy.

The Social Brain Automatically Predicts Others' Future Mental States.

Social life requires people to predict the future: people must anticipate others' thoughts, feelings, and actions to interact with them successfully. The theory of predictive coding suggests that the social brain may meet this need by automatically predicting others' social futures. If so, when representing others' current mental state, the brain should already start representing their future states. To test this hypothesis, we used fMRI to measure female and male human participants' neural representations of mental states. Representational similarity analysis revealed that neural patterns associated with mental states currently under consideration resembled patterns of likely future states more so than patterns of unlikely future states. This effect manifested in activity across the social brain network and in medial prefrontal cortex in particular. Repetition suppression analysis also supported the social predictive coding hypothesis: considering mental states presented in predictable sequences reduced activity in the precuneus relative to unpredictable sequences. In addition to demonstrating that the brain makes automatic predictions of others' social futures, the results also demonstrate that the brain leverages a 3D representational space to make these predictions. Proximity between mental states on the psychological dimensions of rationality, social impact, and valence explained much of the association between state-specific neural pattern similarity and state transition likelihood. Together, these findings suggest that the way the brain represents the social present gives people an automatic glimpse of the social future.SIGNIFICANCE STATEMENT When you see a ball in flight, your brain calculates, not just its static visual features such as size and shape, but also predicts its future trajectory. Here, we investigated whether the same might hold true in the social world: when we see someone flying into a rage, does our brain automatically predict their social trajectory? In this study, we scanned participants' brain activity while they judged others' mental states. We found that neural activity associated with a given state resembled activity associated with likely future states. Additionally, unpredictable sequences of states evoked more brain activity than predictable sequences, consistent with monitoring for, and updating from, prediction errors. These results suggest that the social brain automatically predicts others' future mental states.

The functional effects of prior motion imagery and motion perception

The functional sensory effects and commonalties between mental imagery of different visual features such as color, form or motion remains largely unknown. Mental imagery of static visual features, including color and orientation, can have a facilitative, priming effect on subsequent perception. However, whether motion imagery can have a similar effect remains unknown. Here we used the binocular rivalry method as a measure of motion mental imagery. After imagining or viewing motion of a particular direction, participants were required to report the dominant motion direction in a brief motion rivalry stimulus. We found that motion imagery can have a facilitative priming effect on subsequent motion rivalry perception, and this effect can be attenuated by concurrent expanding and contracting perceptual motion, but not by static or flickering uniform luminance. Unlike color or orientation imagery, the effect of motion imagery on subsequent rivalry was location independent. We also observed this facilitative priming effect with prior low-contrast perceptual motion, but prior high-contrast motion induced a suppressive effect. Simultaneous imagery and perceptual motion in opposite directions induced priming, while congruent directions did not. Counter to prior findings, these results suggest motion imagery can have a priming effect on perception and that the rivalry method can be used to assess visual motion imagery. These results provide evidence for visual imagery as a multi-feature structure.

Wildfire smoke detection based on local extremal region segmentation and surveillance

A novel video-based method is proposed for long-distance wildfire smoke detection. Since the long-distance wildfire smoke usually moves slowly and lacks salient features in the video, the detection is still a challenging problem. Unlike many traditional video-based methods that usually rely on the smoke color or motion for initial smoke region segmentation, we use the Maximally Stable Extremal Region (MSER) detection method to extract local extremal regions of the smoke. This makes the initial segmentation of possible smoke region less dependent on the motion and color information. Potential smoke regions are then selected from all the possible regions by using some static visual features of the smoke, helping to eliminate the non-smoke regions as many as possible. Once a potential smoke region is found, we keep tracking it by searching the best-matched extremal regions in the subsequent frames. At the same time, the propagating motions of the potential smoke region are monitored based on a novel cumulated region approach, which can be effectively used to identify the distinctive expanding and rising motions of smoke. This approach can also make the final smoke motion identification insensitive to image shaking. It was proved that the proposed method is able to reliably detect the long-distance wildfire smoke and simultaneously produce very few false alarms in actual applications.

Differences in Visual-Spatial Input May Underlie Different Compression Properties of Firing Fields for Grid Cell Modules in Medial Entorhinal Cortex.

Firing fields of grid cells in medial entorhinal cortex show compression or expansion after manipulations of the location of environmental barriers. This compression or expansion could be selective for individual grid cell modules with particular properties of spatial scaling. We present a model for differences in the response of modules to barrier location that arise from different mechanisms for the influence of visual features on the computation of location that drives grid cell firing patterns. These differences could arise from differences in the position of visual features within the visual field. When location was computed from the movement of visual features on the ground plane (optic flow) in the ventral visual field, this resulted in grid cell spatial firing that was not sensitive to barrier location in modules modeled with small spacing between grid cell firing fields. In contrast, when location was computed from static visual features on walls of barriers, i.e. in the more dorsal visual field, this resulted in grid cell spatial firing that compressed or expanded based on the barrier locations in modules modeled with large spacing between grid cell firing fields. This indicates that different grid cell modules might have differential properties for computing location based on visual cues, or the spatial radius of sensitivity to visual cues might differ between modules.

Multi-feature hierarchical topic models for human behavior recognition

Human behavior recognition is one important task of image processing and surveillance system. One main challenge of human behavior recognition is how to effectively model behaviors on condition of unconstrained videos due to tremendous variations from camera motion, background clutter, object appearance and so on. In this paper, we propose two novel Multi-Feature Hierarchical Latent Dirichlet Allocation models for human behavior recognition by extending the bag-of-word topic models such as the Latent Dirichlet Allocation model and the Multi-Modal Latent Dirichlet Allocation model. The two proposed models with three hierarchies including low-level visual features, feature topics, and behavior topics can effectively fuse two different types of features including motion and static visual features, avoid detecting or tracking the motion objects, and improve the recognition performance even if the features are extracted with a great amount of noise. Finally, we adopt the variational EM algorithm to learn the parameters of these models. Experiments on the YouTube dataset demonstrate the effectiveness of our proposed models.

Where to Look Next? Combining Static and Dynamic Proto-objects in a TVA-based Model of Visual Attention

To decide “Where to look next ?” is a central function of the attention system of humans, animals and robots. Control of attention depends on three factors, that is, low-level static and dynamic visual features of the environment (bottom-up), medium-level visual features of proto-objects and the task (top-down). We present a novel integrated computational model that includes all these factors in a coherent architecture based on findings and constraints from the primate visual system. The model combines spatially inhomogeneous processing of static features, spatio-temporal motion features and task-dependent priority control in the form of the first computational implementation of saliency computation as specified by the “Theory of Visual Attention” (TVA, [7]). Importantly, static and dynamic processing streams are fused at the level of visual proto-objects, that is, ellipsoidal visual units that have the additional medium-level features of position, size, shape and orientation of the principal axis. Proto-objects serve as input to the TVA process that combines top-down and bottom-up information for computing attentional priorities so that relatively complex search tasks can be implemented. To this end, separately computed static and dynamic proto-objects are filtered and subsequently merged into one combined map of proto-objects. For each proto-object, attentional priorities in the form of attentional weights are computed according to TVA. The target of the next saccade is the center of gravity of the proto-object with the highest weight according to the task. We illustrate the approach by applying it to several real world image sequences and show that it is robust to parameter variations.

The unassisted visual system on earth and in space

Chuck Oman has been a guide and mentor for research in human perception and performance during space exploration for over 25 years. His research has provided a solid foundation for our understanding of how humans cope with the challenges and ambiguities of sensation and perception in space. In many of the environments associated with work in space the human visual system must operate with unusual combinations of visual and other perceptual cues. On Earth physical acceleration cues are normally available to assist the visual system in interpreting static and dynamic visual features. Here we consider two cases where the visual system is not assisted by such cues. Our first experiment examines perceptual stability when the normally available physical cues to linear acceleration are absent. Our second experiment examines perceived orientation when there is no assistance from the physically sensed direction of gravity. In both cases the effectiveness of vision is paradoxically reduced in the absence of physical acceleration cues. The reluctance to rely heavily on vision represents an important human factors challenge to efficient performance in the space environment.

Video Analysis of Human Gait and Posture to Determine Neurological Disorders

This paper investigates the application of digital image processing techniques to the detection of neurological disorder. Visual information extracted from the postures and movements of a human gait cycle can be used by an experienced neurologist to determine the mental health of the person. However, the current visual assessment of diagnosing neurological disorder is based very much on subjective observation, and hence the accuracy of diagnosis heavily relies on experience. Other diagnostic techniques employed involve the use of imaging systems which can only be operated under highly constructed environment. A prototype has been developed in this work that is able to capture the subject's gait on video in a relatively simple setup, and from which to process the selected frames of the gait in a computer. Based on the static visual features such as swing distances and joint angles of human limbs, the system identifies patients with Parkinsonism from the test subjects. To our knowledge, it is the first time swing distances are utilized and identified as an effective means for characterizing human gait. The experimental results have shown a promising potential in medical application to assist the clinicians in diagnosing Parkinsonism.

Comparison of Image Transform-Based Features for Visual Speech Recognition in Clean and Corrupted Videos

We present results of a study into the performance of a variety of different image transform-based feature types for speaker-independent visual speech recognition of isolated digits. This includes the first reported use of features extracted using a discrete curvelet transform. The study will show a comparison of some methods for selecting features of each feature type and show the relative benefits of both static and dynamic visual features. The performance of the features will be tested on both clean video data and also video data corrupted in a variety of ways to assess each feature type's robustness to potential real-world conditions. One of the test conditions involves a novel form of video corruption we call jitter which simulates camera and/or head movement during recording.

Comparison between different feature extraction techniques for audio-visual speech recognition

The current audio-only speech recognition still lacks the expected robustness when the Signal to Noise Ratio (SNR) decreases. The video information is not affected by noise which makes it an ideal candidate for data fusion for speech recognition benefit. In the paper [1] the authors have shown that most of the techniques used for extraction of static visual features result in equivalent features or at least the most informative features exhibit this property. We argue that one of the main problems of existing methods is that the resulting features contain no information about the motion of the speaker’s lips. Therefore, in this paper we will analyze the importance of motion detection for speech recognition. For this we will first present the Lip Geometry Estimation(LGE) method for static feature extraction. This method combines an appearance based approach with a statistical based approach for extracting the shape of the mouth. The method was introduced in [2] and explored in detail in [3]. Further more, we introduce a second method based on a novel approach that captures the relevant motion information with respect to speech recognition by performing optical flow analysis on the contour of the speaker’s mouth. For completion, a middle way approach is also analyzed. This third method considers recovering the motion information by computing the first derivatives of the static visual features. All methods were tested and compared on a continuous speech recognizer for Dutch. The evaluation of these methods is done under different noise conditions. We show that the audio-video recognition based on the true motion features, namely obtained by performing optical flow analysis, outperforms the other settings in low SNR conditions.

Gamma oscillations and object processing in the infant brain.

An enduring controversy in neuroscience concerns how the brain "binds" together separately coded stimulus features to form unitary representations of objects. Recent evidence has indicated a close link between this binding process and 40-hertz (gamma-band) oscillations generated by localized neural circuits. In a separate line of research, the ability of young infants to perceive objects as unitary and bounded has become a central focus for debates about the mechanisms of perceptual development. Here we demonstrate that binding-related 40-hertz oscillations are evident in the infant brain around 8 months of age, which is the same age at which behavioral and event-related potential evidence indicates the onset of perceptual binding of spatially separated static visual features.

Acquisition of behavioral control by static visual features of an imprinting object: Species generality

Previous work had indicated that after prolonged exposure to a moving object, ducklings will direct filial responses toward that object even if it remains stationary during its presentation. The present experiment demonstrated the wide species generality of this effect by obtaining similar results in newly hatched chickens, pheasant, turkeys, and quail, as well as ducks.

Development of filial attachment to static visual features of an imprinting object

Three experiments investigated the fact that the static visual features of an imprinting object (its color, size, and shape characteristics, as represented by the object remaining stationary during its presentation) can gradually acquire control over filial behavior by virtue of their repeated spatial-temporal pairing with visual motion—an effect that has been interpreted as a process of classical conditioning. Experiment 1 found that the static features can acquire control if they are made conspicuous in some other, nonmovement way (i.e., by manipulating the relative illumination of the imprinting object). Further experiments then failed to find any qualitative difference between the behavioral control developed by conspicuous but static aspects of a duckling’s environment and the control developed by the static features of a moving object, either in terms of persistence of the acquired control (Experiment 2) or in terms of possible restrictions imposed by the sensitive period for imprinting (Experiment 3). Taken together, these results support the plausibility of a perceptual learning hypothesis, although the classical conditioning view was not itself contradicted. The present findings are also relevant to the broader issue of whether any valid distinctions exist between the behavioral control exerted by static aspects of a precocial bird’s environment and the control exerted by animate, presumably social stimuli.

Behavioral control by stimulus components of an imprinting object

In the first of three experiments, ducklings that had received prolonged exposure to the visibly moving imprinting object subsequently suppressed ongoing distress vocalization both during brief presentations of the moving object and during brief presentations of its initially neutral stimulus components (i.e., its auditory and static visual features). Only presentations of the moving object were followed by priming aftereffects (namely, enhancement of distress vocalization over a baseline rate). In Experiment 2, weak, but reliable, priming effects were detected after very long presentations of the auditory and static visual features. Experiment 3 found that these features strongly suppressed low, but not high, rates of distress vocalization, while the visibly moving object strongly suppressed both high and low rates. These studies suggest that initially neutral features of an imprinting object acquire the same sort of behavioral control as is exerted by the object when it is in motion, but that this control is somewhat weaker.

PERSISTENCE OF ACQUIRED BEHAVIORAL CONTROL IN THE CONTEXT OF IMPRINTING1

Newly hatched Khaki Campbell ducklings (Anas platyrhynchos domesticus) were exposed to a moving object that immediately suppressed distress vocalizations occurring in a novel environment. The static visual and auditory features of this object acquired the ability to suppress distress vocalizations after eight 20-min sessions of exposure to the object in motion. The acquired suppressive properties of these features were found to persist throughout thirty 20-min sessions given over 10 days. During these sessions, the ducklings were continually exposed to the static features in the absence of visual movement. In a second experiment, the ability of these features in the absence of visual movement. In a second experiment, the ability of these features to serve as reinforcement for a pecking response was shown to persist for up to 56 hr. In one duckling, presentations of the static visual features did not maintain pecking behavior. However, it was shown that pecking responses could be re-instated in this duckling by introducing novel stimuli to the environment.