Temporal Fusion Research Articles

The Temporal Fusion Illusion and its neurophysiological correlates

The Temporal Fusion Illusion and its neurophysiological correlates

A model for spatial and temporal data fusion

Spatial and temporal fusion of remote sensing data is a technology that can generate dense time series and high spatial resolution data whose spatial resolution is similar with high spatial resolution date,and temporal resolution is the same as the one with high temporal resolution data.This paper presented a new spatial and temporal data fusion model(STDFM) for blending Landsat and MODIS surface reflectance.Temporal change information was detected from sequence coarser resolution surface images,and new high resolution reflectance was predicted from former high resolution reflectance.This algorithm was tested in red and near-infrared MODIS and Landsat ETM+ images,and over a study area in Jiangning country,Nanjing,Jiangsu,China.Results showed that STDFM was able to produce images very similar with actual observed images.The correlation coefficient r between synthetic imageries and actual observations was 0.939.The correlation coefficient r between NDVI calculated by synthetic imageries and actual observations was 0.938.

A Decision Fusion and Reasoning Module for a Traffic Sign Recognition System

A novel approach for a decision fusion and reasoning system for vision-based traffic sign recognition is presented. This module consists of several steps. In the first stage, a track-based Bayesian fusion scheme is used to fuse the classification results from each frame to obtain a fusion result for each track to decide whether a sign is present, as well as to determine the sign type. In order to determine the sign type, the temporal fusion scheme has been combined with a decision tree. In the second stage, the system combines and fuses probable identical objects which help to further reduce failures in the recognition process. The decision is based on the fusion results, as well as a position cue. Finally, a reasoning module is used to decide which of the passed signs should be shown to the driver. In addition to these modules, a general evaluation method for multi-class tracking systems is shown. While some failures are observed from the evaluation on object level, the additional post processing steps improve the system in such a way that the finally presented signs are almost always correct on the test set.

Video fusion performance evaluation based on structural similarity and human visual perception

In order to evaluate different video fusion algorithms in temporal stability and consistency as well as in spatial information transfer, a novel objective video fusion quality metric is proposed with the structural similarity (SSIM) index and the perception characteristics of human visual system (HVS) in this paper. Firstly, for each frame, two sub-indices, i.e., the spatial fusion quality index and the temporal fusion quality index, are defined by the weighted local SSIM indices. Secondly, for the current frame, an individual-frame fusion quality measure is obtained by integrating the above two sub-indices. Lastly, the proposed global video fusion metric is constructed as the weighted average of all the individual-frame fusion quality measures. In addition, according to the perception characteristics of HVS, some local and global spatial–temporal information, such as local variance, pixel movement, global contrast, background motion and so on, is employed to define the weights in the proposed metric. Several sets of experimental results demonstrate that the proposed metric can evaluate different video fusion algorithms accurately, and the evaluation results coincide with the subjective results well.

Odor identity influences tracking of temporally patterned plumes in Drosophila

BackgroundTurbulent fluid landscapes impose temporal patterning upon chemical signals, and the dynamical neuronal responses to patterned input vary across the olfactory receptor repertoire in flies, moths, and locusts. Sensory transformations exhibit low pass filtering that ultimately results in perceptual fusion of temporally transient sensory signals. For example, humans perceive a sufficiently fast flickering light as continuous, but the frequency threshold at which this fusion occurs varies with wavelength. Although the summed frequency sensitivity of the fly antenna has been examined to a considerable extent, it is unknown how intermittent odor signals are integrated to influence plume tracking behavior independent of wind cues, and whether temporal fusion for behavioral tracking might vary according to the odor encountered.ResultsHere we have adopted a virtual reality flight simulator to study the dynamics of plume tracking under different experimental conditions. Flies tethered in a magnetic field actively track continuous (non-intermittent) plumes of vinegar, banana, or ethyl butyrate with equal precision. However, pulsing these plumes at varying frequency reveals that the threshold rate, above which flies track the plume as if it were continuous, is unique for each odorant tested. Thus, the capability of a fly to navigate an intermittent plume depends on the particular odorant being tracked during flight. Finally, we measured antennal field potential responses to an intermittent plume, found that receptor dynamics track the temporal pattern of the odor stimulus and therefore do not limit the observed behavioral temporal fusion limits.ConclusionsThis study explores the flies' ability to track odor plumes that are temporally intermittent. We were surprised to find that the perceptual critical fusion limit, determined behaviorally, is strongly dependent on odor identity. Antennal field potential recordings indicate that peripheral processing of temporal cues faithfully follow rapid odor transients above the rates that can be resolved behaviorally. These results indicate that (1) higher order circuits create a perceptually continuous signal from an intermittent sensory one, and that (2) this transformation varies with odorant rather than being constrained by sensory-motor integration, thus (3) offering an entry point for examining the mechanisms of rapid olfactory decision making in an ecological context.

Eyebrow Peak Position Redefined

The aesthetically appealing eyebrow shape has been defined by its arch, located near the junction between the medial two-thirds and lateral one-third. The position of this arch has been historically described by arbitrary anatomical landmarks that have no logical structural relationship. Moreover, selection of endoscopic brow lift incision sites that define vector of pull and fixation points have been variably described. The authors examine the position of the deep temporal fusion line to determine whether it can act as a more accurate and functional landmark than prior anatomical landmarks for the eyebrow peak position. Eyebrows were measured in 50 subjects from the medial aspect of the eyebrow to the a) deep temporal fusion line (ridge), b) eyebrow peak (arch), c) lateral aspect of the brow, and d) lateral limbus. Pearson's correlation, descriptive statistics, and student's t test results were obtained. Eyebrow measurements demonstrated that the deep temporal fusion line is the most precise indicator of brow peak position among all examined landmarks. The Pearson correlation value was strongest between brow peak and deep temporal fusion line (P = .860) and a t test confirmed this observation with no significant difference between brow peak and deep temporal fusion line. The lateral limbus and medial two-thirds lateral one-third junction more accurately predict brow peak in females, but the deep temporal fusion line is an equally reliable predictor of brow peak for males and females. These findings suggest that placement of endoscopic brow lift incisions and subsequent fixation points may be best defined along the deep temporal fusion line.

Lane Detection and Tracking Based on Lidar Data

The contribution presents a novel approach to the dete ction and tracking of lanes based on lidar data. Therefore, we use the distance and reflectivity data coming from a one-dimensional sensor. After having detected the lane through a temporal fusion algorithm, we register the l idar data in a world-fixed coordinate system. To this end, we also incorporate the data coming from an inertial measurement unit and a differential global positioning system. After that stage, an original image of the roa d can be inferred. Based on this data view, we are able to track the lane either with a Kalman filter or by using a polynomial approximation for the underlying lane model.

Multi-agent coordination by temporal plan fusion: Application to combat search and rescue

Coordination of actions and plans that must be achieved by multiple agents is one of the most difficult tasks in the multi-agent domain. In order to work together and achieve a common goal, agents need to coordinate their plans in a way that guarantees, if possible, the success of each individual agent plan. In this work, we propose a temporal fusion mechanism that allows a set of agents to fuse their plans and generate a global coordinated plan. First, we define a temporal plan as a set of temporally constrained actions. The fusion of several temporal plans is a temporal plan, which can be executed by several agents. The proposed framework is applied to a Combat Search and Rescue application.

Spatial grouping determines temporal integration.

To make sense out of a continuously changing visual world, people need to integrate features across space and time. Despite more than a century of research, the mechanisms of features integration are still a matter of debate. To examine how temporal and spatial integration interact, the authors measured the amount of temporal fusion (a measure of temporal integration) for different spatial layouts. They found that spatial grouping by proximity and similarity can completely block temporal integration. Computer simulations with a simple neural network capture these findings very well, suggesting that the proposed spatial grouping operations may occur already at an early stage of visual information processing.

Spatial, Temporal, and Interchannel Image Data Fusion for Long-Distance Terrestrial Observation Systems

This paper presents methods for intrachannel and interchannel fusion of thermal and visual sensors used in long-distance terrestrial observation systems. Intrachannel spatial and temporal fusion mechanisms used for image stabilization, super-resolution, denoising, and deblurring are supplemented by interchannel data fusion of visual- and thermal-range channels for generating fused videos intended for visual analysis by a human operator. Tests on synthetic, as well as on real-life, video sequences have confirmed the potential of the suggested methods.

Local Vote Decision Fusion for Target Detection in Wireless Sensor Networks

This paper examines the problem of target detection by a wireless sensor network. Sensors acquire measurements emitted from the target that are corrupted by noise, and initially make individual decisions about the presence/absence of the target. We propose the local vote decision fusion algorithm, in which sensors first correct their decisions using decisions of neighboring sensors, and then make a collective decision as a network. An explicit formula that approximates the system's decision threshold for a given false alarm rate is derived using limit theorems for random fields, which provides a theoretical performance guarantee for the algorithm. We examine both distance- and nearest-neighbor-based versions of the local vote algorithm for grid and random sensor deployments and show that, in many situations, for a fixed-system false alarm, the local vote correction achieves significantly higher target detection rate than decision fusion based on uncorrected decisions. The algorithm does not depend on the signal model and is shown to be robust to different types of signal decay. We also extend this framework to temporal fusion, where information becomes available over time.

Temporal Bayesian Network based contextual framework for structured information mining

Specific domains in video data contain rich temporal structures that help in classification process. In this paper, we exploit the temporal structure to characterize video sequence data into different classes. We propose the following perceptual features: Time-to-Collision, shot length and transition, and temporal motion activity. Using these perceptual features, several video classes are characterized leading to formation of high-level sequence classification. Resulting high-level queries are more easily mapped onto the perceptual features enabling better accessibility of content-based retrieval systems. Temporal fusion of the perceptual features forms higher-level structures, which can be effectively tackled using the Dynamic Bayesian Networks. The Networks allow the power of statistical inference and learning to be combined with the temporal and contextual knowledge of the problem. The modeling and experimental results are presented for a number of key applications, like sequence identification, extracting highlights for sports, and parsing a news program.

Using dynamic time warping for online temporal fusion in multisensor systems

Sensor fusion is concerned with gaining information from multiple sensors by fusing across raw data, features or decisions. Traditionally these fusion processes only concern fusion at specific points in time. However recently, there is a growing interest in inferring the behavioural aspects of environments or objects that are monitored by multisensor systems, rather than just their states at specific points in time. In order to infer environmental behaviours, it may be necessary to fuse data acquired from (i) geographically distributed sensors at specific points of time and (ii) specific sensors over a period of time. Fusing multisensor data over a period of time (also known as Temporal fusion) is a challenging task, since the data to be fused consists of complex sequences that are multi-dimensional, multimodal, interacting, and time-varying in nature. Additionally, performing temporal fusion efficiently in real-time is another challenge due to the large amounts of data to be fused. To address this issue, we propose a robust and efficient framework that uses dynamic time warping (DTW) as the core recognizer to perform online temporal fusion on either the raw data or the features. We evaluate the performance of the online temporal fusion system on two real world datasets: (1) accelerometer data acquired from performing two hand gestures, and (2) a benchmark dataset acquired from carrying a mobile device and performing the predefined user scenarios. Performance results of the DTW-based system are compared with those of a Hidden Markov Model (HMM) based system. The experimental results from both datasets demonstrate that the proposed system outperforms HMM based systems, and has the capability to perform online temporal fusion efficiently and accurately in real-time.

Ontological inference for image and video analysis

This paper presents an approach to designing and implementing extensible computational models for perceiving systems based on a knowledge-driven joint inference approach. These models can integrate different sources of information both horizontally (multi-modal and temporal fusion) and vertically (bottom–up, top–down) by incorporating prior hierarchical knowledge expressed as an extensible ontology.Two implementations of this approach are presented. The first consists of a content-based image retrieval system that allows users to search image databases using an ontological query language. Queries are parsed using a probabilistic grammar and Bayesian networks to map high-level concepts onto low-level image descriptors, thereby bridging the ‘semantic gap’ between users and the retrieval system. The second application extends the notion of ontological languages to video event detection. It is shown how effective high-level state and event recognition mechanisms can be learned from a set of annotated training sequences by incorporating syntactic and semantic constraints represented by an ontology.

Fusion of multisensor multitemporal satellite data for land cover mapping

Efficient integration of remote sensing information with different temporal, spectral and spatial resolutions is important for accurate land cover mapping. A new temporal fusion classification (TFC) model is presented for land cover classification, based on statistical fusion of multitemporal satellite images. In the proposed model, the temporal dependence of multitemporal images is taken into account by estimating transition probabilities from the change pattern of a vegetation dynamics indicator (VDI). Extension of this model is applicable to Synthetic Aperture Radar (SAR) images and integration of multisensor multitemporal satellite images, concerning both temporal attributes and reliability of multiple data sources. The feasibility of the new method is verified using multitemporal Landsat Thematic Mapper (TM) and ERS SAR satellite images, and experimental results show improved performance over conventional methods.

Research of Spatial-Temporal Architecture Model and the Algorithm for Multisensor Information Fusion

To receive habitus accurately and to obtain the reorganization and decision information of the object fully, spatial and temporal fusion must be taken into account. In this paper, a spatial-temporal two-layer architecture model is presented. In this architecture, the data monitored by different sensors at the same time are fused first in space, and then in time order. Based on this model, we give an algorithm based on the theory of D-S evidence and fuzzy integral for sensor fusion. Simulation results indicate that this algorithm can improve the efficiency of object identification and the reliability of system. The simulation of the algorithm in multisensor object identification is also presented.

Motion detection and tracking using belief indicators for an automatic visual-surveillance system

A motion detection and tracking algorithm for human and car activity surveillance is presented and evaluated by using the Pets'2000 test sequence. The proposed approach uses a temporal fusion strategy by using the history of events in order to improve instantaneous decisions. Normalized indicators updated at each frame summarize the history of specific events. For the motion detection stage, a fast updating algorithm of the background reference is proposed. The control of the updating at each pixel is based on a stability indicator estimated from inter-frame variations. The tracking algorithm uses a region-based approach. A belief indicator representing the tracking consistency for each object allows solving defined ambiguities at the tracking level. A second specific tracking indicator representing the identity quality of each tracked object is updated by integrating object interaction. Tracking indicators permit to propagate uncertainties on higher levels of the interpretation and are directly useful in the tracking performance evaluation.

Sparse Bayesian learning for efficient visual tracking

This paper extends the use of statistical learning algorithms for object localization. It has been shown that object recognizers using kernel-SVMs can be elegantly adapted to localization by means of spatial perturbation of the SVM. While this SVM applies to each frame of a video independently of other frames, the benefits of temporal fusion of data are well-known. This is addressed here by using a fully probabilistic Relevance Vector Machine (RVM) to generate observations with Gaussian distributions that can be fused over time. Rather than adapting a recognizer, we build a displacement expert which directly estimates displacement from the target region. An object detector is used in tandem, for object verification, providing the capability for automatic initialization and recovery. This approach is demonstrated in real-time tracking systems where the sparsity of the RVM means that only a fraction of CPU time is required to track at frame rate. An experimental evaluation compares this approach to the state of the art showing it to be a viable method for long-term region tracking.

Classification of bioacoustic time series based on the combination of global and local decisions

Automated classification of cricket songs from Thailand and Ecuador is the topic of this study. For this, the locations of pulses are determined and different features in the time and the frequency domain are extracted automatically from the time series. For the categorization of the sound patterns these features are combined through data fusion, temporal fusion and decision fusion. Local features and global features are distinguished. For the classification a fuzzy- k-nearest-neighbour classifier was used. Classification results for a data set containing songs of 28 different species are presented.

Adaptive spatiotemporal multiple sensor fusion

We have developed and applied a spatiotemporal fusion framework that uses different fusion strategies across time frames (temporal fusion) as well as between sensors (spatial fusion). We have developed, at the feature level, new and different fusion strategies (additive and minmax fusion) in addition to the traditional strategies (multiplicative, min, and max fusion). These different fusion strategies are compared and predicted by their receiver operating characteristics performance in the likelihood-reading domain. Furthermore, by applying the additive fusion strategy, we have developed methods for adaptive sensor weighting using reliability functions to improve fusion performance. Our simulated test and analysis results show that temporal fusion (as well as spatial fusion) can considerably improve target classification, and also show that the adaptive sensor weighting using reliability functions can significantly improve fusion performance when one sensor is much more reliable than the other. Finally, we propose an optimal integrated spatiotemporal multiple sensor fusion system that includes two new processors: the adaptive processor and the fusion selection processor.