Wide Area Motion Imagery Research Articles

Wide Area Motion Imagery and the Colonial Antecedents of Surveillance

Wide Area Motion Imagery and the Colonial Antecedents of Surveillance

Crowd-aware Thresholded Loss for Object Detection in Wide Area Motion Imagery

Crowd-aware Thresholded Loss for Object Detection in Wide Area Motion Imagery

HMRN: heat map regression network to detect and track small objects in wide-area motion imagery

HMRN: heat map regression network to detect and track small objects in wide-area motion imagery

Online real-time pedestrian tracking from medium altitude aerial footage with camera motion cancellation

Using medium altitude footage in various image based pedestrian detection and tracking tasks have several advantages as opposed to low altitude footage. For instance, much larger covered area, or weaker perspective distortions and occlusions. Furthermore, it does not pose privacy issues. However, it also has a disadvantage that is the irregular camera motion. Such motions can be observed with some ground level footage as well. Some state-of-the-art multi-object tracking (MOT) algorithms, which excel at pedestrian tracking, handle these implicitly, however this is insufficient in the case of medium altitude footage. The problem is further enhanced in online real-time tracking where frames may need to be skipped. Although there are explicit frame registration methods developed for wide area motion imagery (WAMI), they operate on assumptions that are not true in medium altitude footage such as the background is mostly stationary. We show that by augmenting the pedestrian tracking algorithms with a simple visual object tracking (VOT) based frame registration method their accuracy on medium altitude footage can be substantially improved without hindering the execution speed. Moreover, we show that the algorithms’ execution speed can be significantly increased by better utilization of the resources. To evaluate these, we introduce helicopter video sequences with MOT annotation.

HM-Net: A Regression Network for Object Center Detection and Tracking on Wide Area Motion Imagery

Wide Area Motion Imagery (WAMI) yields high resolution images with a large number of extremely small objects. Target objects have large spatial displacements throughout consecutive frames. This nature of WAMI images makes object tracking and detection challenging. In this paper, we present our deep neural network-based combined object detection and tracking model, namely, Heat Map Network (HM-Net). HM-Net is significantly faster than state-of-the-art frame differencing and background subtraction-based methods, without compromising detection and tracking performances. HM-Net follows object center-based joint detection and tracking paradigm. Simple heat map-based predictions support unlimited number of simultaneous detections. The proposed method uses two consecutive frames and the object detection heat map obtained from the previous frame as input, which helps HM-Net monitor spatio-temporal changes between frames and keeps track of previously predicted objects. Although reuse of prior object detection heat map acts as a vital feedback-based memory element, it can lead to unintended surge of false positive detections. To increase robustness of the method against false positives and to eliminate low confidence detections, HM-Net employs novel feedback filters and advanced data augmentations. HM-Net outperforms state-of-the-art WAMI moving object detection and tracking methods on WPAFB dataset with its 96.2% F1 and 94.4% mAP detection scores, while achieving a 61.8% mAP tracking score on the same dataset.

DCT-Based Local Descriptor for Robust Matching and Feature Tracking in Wide Area Motion Imagery

We introduce a novel discrete cosine transform-based feature (DCTF) descriptor designed for both robustly matching features in aerial video and tracking features across wide-baseline oblique views in aerial wide area motion imagery (WAMI). Our DCTF descriptor preserves local structure more compactly in the frequency domain by utilizing the mathematical properties of the discrete cosine transform (DCT) and outperforms widely used the spatial-domain feature extraction methods, such as speeded up robust features (SURF) and scale-invariant feature transform (SIFT). The DCTF descriptor can be used in combination with other feature detectors, such as SURF and features from accelerated segment test (FAST), for which we provide experimental results. The performance of DCTF for image matching and feature tracking is evaluated on two city-scale aerial WAMI data sets (ABQ-215 and LA-351) and a synthetic aerial drone video data set digital imaging and remote sensing image generation (Rochester Institute of Technology (RIT)-DIRSIG). DCTF is a compact 120-D descriptor that is less than half the dimensionality of state-of-the-art deep learning-based approaches, such as SuperPoint, LF-Net, and DeepCompare, which requires no learning and is domain-independent. Despite its small size, the DCTF descriptor surprisingly produces the highest image matching accuracies ( F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> = 0.76 and ABQ-215), the longest maximum and average feature track lengths, and the lowest tracking error (0.3 pixel, LA-351) compared with both handcrafted and deep learning features.

OBJECT DETECTION UNDER MOVING CLOUD SHADOWS IN WAMI

Abstract. For a reliable and robust moving object detection, the subtraction of a precisely modeled background is crucial in wide-area motion imagery (WAMI). Even the most successful background subtraction algorithms that are designed to model highly-dynamic environments cannot cope with rapidly changing scenery, such as moving cloud shadows, which has different characteristics from dynamic textures. This paper presents a novel method to detect moving objects and to eliminate false alarms under moving cloud shadow regions in gray-level video sequences. The proposed method uses the relation between reflectance values of the shadowed and well-illuminated sequences of the regions in the video frame. A modified adaptive region growing approach, which extends from seed points, is designed to obtain the moving parts of the cloud shadows without presuming the geometric structure of the clouds. In order to determine the moving border of the cloud shadows, where false alarms typically occur, the cloud shadow motion should be detected. As the last stage of the proposed method, real moving objects in the scene are tried to be discriminated from false alarms by exploiting the relation of intensity ratios between the object candidate and its surroundings. The accuracy and computational efficiency of the proposed approach make it a reliable and feasible approach to be used in real-time surveillance solutions.

Multiple Hypothesis Tracking with Kinematics and Appearance Models on Traffic Flow for Wide Area Traffic Surveillance

AbstractThis paper presents a study of multiple hypothesis tracking (MHT) of vehicles recorded in wide area motion imagery (WAMI) that has persistent coverage. To take advantage of visual informati...

Tracking in Aerial Hyperspectral Videos Using Deep Kernelized Correlation Filters

Hyperspectral imaging holds enormous potential to improve the state of the art in aerial vehicle tracking with low spatial and temporal resolutions. Recently, adaptive multimodal hyperspectral sensors have attracted growing interest due to their ability to record extended data quickly from aerial platforms. In this paper, we apply popular concepts from traditional object tracking, namely, kernelized correlation filters (KCFs) and deep convolutional neural network features to aerial tracking in the hyperspectral domain. We propose the deep hyperspectral KCF-based tracker (DeepHKCF) to efficiently track aerial vehicles using an adaptive multimodal hyperspectral sensor. We address low temporal resolution by designing a single KCF-in-multiple regions-of-interest (ROIs) approach to cover a reasonably large area. To increase the speed of deep convolutional features extraction from multiple ROIs, we design an effective ROI mapping strategy. The proposed tracker also provides flexibility to couple with the more advanced correlation filter trackers. The DeepHKCF tracker performs exceptionally well with deep features set up in a synthetic hyperspectral video generated by the digital imaging and remote sensing image generation (DIRSIG) software. In addition, we generate a large, synthetic, single-channel data set using DIRSIG to perform vehicle classification in the wide-area motion imagery (WAMI) platform. This way, the high fidelity of the DIRSIG software is proven, and a large-scale aerial vehicle classification data set is released to support studies on vehicle detection and tracking in the WAMI platform.

Aerial Wide-Area Motion Imagery Registration Using Automated Multiscale Feature Selection

Automatic registration of aerial wide-area motion imagery is required to correct the camera parameters in order to eliminate the geocoding errors arising from frequent reinstallation of the camera array on the carrier aircraft. Approaches developed to date solely rely on the information present in the imagery not using a priori knowledge about the environment and the features present in it in the sequence analysis. To this end, we propose a novel method based on dynamic feature extraction and automatic multiscale feature matching to produce per-frame camera pose corrections. The features are extracted from the imagery using one of the three dedicated classifiers and are then robustly matched to the features projected from the datum using a coarse-to-fine multiscale approach. Finally, the bias between the estimated and the actual camera pose is estimated using ordinary least squares optimization performed on the distances between the approved match candidate pairs. The application of the proposed method to 50 frames of very high-resolution aerial imagery captured over mixed terrain at an altitude of 5.3 km demonstrates significant reduction in position error of the features (from 47.76 to 12.31 m) and proves the attractiveness of the approach as an alternative to manual labeling methods using ground control points.

Online Data Thinning via Multi-Subspace Tracking.

In an era of ubiquitous large-scale streaming data, the availability of data far exceeds the capacity of expert human analysts. In many settings, such data is either discarded or stored unprocessed in data centers. This paper proposes a method of online data thinning, in which large-scale streaming datasets are winnowed to preserve unique, anomalous, or salient elements for timely expert analysis. At the heart of this proposed approach is an online anomaly detection method based on dynamic, low-rank Gaussian mixture models. Specifically, the high-dimensional covariance matrices associated with the Gaussian components are associated with low-rank models. According to this model, most observations lie near a union of subspaces. The low-rank modeling mitigates the curse of dimensionality associated with anomaly detection for high-dimensional data, and recent advances in subspace clustering and subspace tracking allow the proposed method to adapt to dynamic environments. Furthermore, the proposed method allows subsampling, is robust to missing data, and uses a mini-batch online optimization approach. The resulting algorithms are scalable, efficient, and are capable of operating in real time. Experiments on wide-area motion imagery and e-mail databases illustrate the efficacy of the proposed approach.

Vehicle Tracking in Wide Area Motion Imagery via Stochastic Progressive Association Across Multiple Frames (SPAAM).

Vehicle tracking in Wide Area Motion Imagery (WAMI) relies on associating vehicle detections across multiple WAMI frames to form tracks corresponding to individual vehicles. The temporal window length, i.e., the number M of sequential frames, over which associations are collectively estimated poses a trade-off between accuracy and computational complexity. A larger M improves performance because the increased temporal context enables the use of motion models and allows occlusions and spurious detections to be handled better. The number of total hypotheses tracks, on the other hand, grows exponentially with increasing M, making larger values of M computationally challenging to tackle. In this paper, we introduce SPAAM an iterative approach that progressively grows M with each iteration to improve estimated tracks by exploiting the enlarged temporal context while keeping computation manageable through two novel approaches for pruning association hypotheses. First, guided by a road network, accurately co-registered to the WAMI frames, we disregard unlikely associations that do not agree with the road network. Second, as M is progressively enlarged at each iteration, the related increase in association hypotheses is limited by revisiting only the subset of association possibilities rendered open by stochastically determined dis-associations for the previous iteration. The stochastic disassociation at each iteration maintains each estimated association according to an estimated probability for confidence, obtained via a probabilistic model. Associations at each iteration are then estimated globally over the M frames by (approximately) solving a binary integer programming problem for selecting a set of compatible tracks. Vehicle tracking results obtained over test WAMI datasets indicate that our proposed approach provides significant performance improvements over state of the art alternatives.

Online single target tracking in WAMI: benchmark and evaluation

Online single target tracking (OSTT) is a prominent topic in normal surveillance environments for security and transportation applications. However, OSTT comparative analysis is seriously under-investigated in the context of wide area motion imagery (WAMI) although its importance keeps rising with the popularity of the unmanned aerial vehicles. In this work, we make several efforts toward WAMI tracking analysis. First, we propose a new WAMI OSTT benchmark dataset, named WAMI-226, which consists of 100 image frames and 226 targets. This new benchmark dataset brings together research challenges including low frame rate, low resolution, and low contrast. Second, we evaluate 20 existing online trackers for WAMI tracking scenarios. Third, by combining the basic appearance model, background subtraction and high-order motion (HoM) affinity, we develop a novel normalized cross correlation HoM (NCC-HoM) tracking algorithm for WAMI OSTT. The experimental results show that the proposed NCC-HoM method achieves significant improvements for both target initialization and online tracking. Thus, NCC-HoM serves as a new baseline algorithm for the WAMI-226 benchmark.

COSMOS

Hardware accelerators are key to the efficiency and performance of system-on-chip (SoC) architectures. With high-level synthesis (HLS), designers can easily obtain several performance-cost trade-off implementations for each component of a complex hardware accelerator. However, navigating this design space in search of the Pareto-optimal implementations at the system level is a hard optimization task. We present COSMOS, an automatic methodology for the design-space exploration (DSE) of complex accelerators, that coordinates both HLS and memory optimization tools in a compositional way. First, thanks to the co-design of datapath and memory, COSMOS produces a large set of Pareto-optimal implementations for each component of the accelerator. Then, COSMOS leverages compositional design techniques to quickly converge to the desired trade-off point between cost and performance at the system level. When applied to the system-level design (SLD) of an accelerator for wide-area motion imagery (WAMI), COSMOS explores the design space as completely as an exhaustive search, but it reduces the number of invocations to the HLS tool by up to 14.6×.

Tile prediction schemes for wide area motion imagery maps in GIS

Wide-area surveillance, traffic monitoring, and emergency management are just several of many applications benefiting from the incorporation of Wide-Area Motion Imagery (WAMI) maps into geographic information systems. Though the use of motion imagery as a GIS base map via the Web Map Service (WMS) standard is not a new concept, effectively streaming imagery is particularly challenging due to its large scale and the multidimensionally interactive nature of clients that use WMS. Ineffective streaming from a server to one or more clients can unnecessarily overwhelm network bandwidth and cause frustratingly large amounts of latency in visualization to the user. Seamlessly streaming WAMI through GIS requires good prediction to accurately guess the tiles of the video that will be traversed in the near future. In this study, we present an experimental framework for such prediction schemes by presenting a stochastic interaction model that represents a human user’s interaction with a GIS video map. We then propose several algorithms by which the tiles of the stream may be predicted. Results collected both within the experimental framework and using human analyst trajectories show that, though each algorithm thrives under certain constraints, the novel Markovian algorithm yields the best results overall. Furthermore, we make the argument that the proposed experimental framework is sufficient for the study of these prediction schemes.

Elimination of resampling errors in wide area motion imagery (WAMI)

Extraction of useful information from Wide Area Motion Imagery (WAMI) data is limited by the resolution and clarity of the image. The resolution of the raw image is defined by the camera, optics, and conversion electronics. For use by an analyst, the raw image must be converted to a geo-rectified image. This projection process requires resampling, which results in a loss of resolution. In addition, compression may be required to accommodate a limited band width transmission link or limited storage space, again with a loss of resolution. Resampling and compression can reduce the clarity of the image as seen by the analyst. In this article, the impact of resampling and compression for the Wright Patterson Air Force Base (WPAFB) 20091021 data set are analyzed. This data set contains both raw images and projected images. The projected images are generated using the traditional sensor exploitation tool (SET) model. The raw images are then projected using the SET model developed here. The resolution of the traditional SET and the proposed SET are measured. Measurement of the compression loss shows that compression is not responsible for the loss of resolution. The resolution is limited by the resampling process. To eliminate resampling errors, a novel SET model is proposed. The proposed SET model preserves the resolution of the original raw image by eliminating the resampling process before data storage and transmission.

Deterministic object tracking using Gaussian ringlet and directional edge features

Challenges currently existing for intensity-based histogram feature tracking methods in wide area motion imagery (WAMI) data include object structural information distortions, background variations, and object scale change. These issues are caused by different pavement or ground types and from changing the sensor or altitude. All of these challenges need to be overcome in order to have a robust object tracker, while attaining a computation time appropriate for real-time processing. To achieve this, we present a novel method, Directional Ringlet Intensity Feature Transform (DRIFT), which employs Kirsch kernel filtering for edge features and a ringlet feature mapping for rotational invariance. The method also includes an automatic scale change component to obtain accurate object boundaries and improvements for lowering computation times. We evaluated the DRIFT algorithm on two challenging WAMI datasets, namely Columbus Large Image Format (CLIF) and Large Area Image Recorder (LAIR), to evaluate its robustness and efficiency. Additional evaluations on general tracking video sequences are performed using the Visual Tracker Benchmark and Visual Object Tracking 2014 databases to demonstrate the algorithms ability with additional challenges in long complex sequences including scale change. Experimental results show that the proposed approach yields competitive results compared to state-of-the-art object tracking methods on the testing datasets.

A Monte Carlo tradeoff analysis to guide resource investment in threat detection systems: From forensic to prospective investigations

The Department of Defense has invested in threat detection systems, consisting of sensors and algorithms to collect and process data on potential threats. We performed a Monte Carlo tradeoff analysis to determine the optimal combination of sensor versus algorithm capability for an airborne wide area motion imagery (WAMI) system used to identify insurgents detonating improvised explosive devices (IEDs) in urban environments. We created a computational model of the WAMI system, including the camera and algorithms for processing, exploitation, and dissemination. We also synthesized input data consisting of objects and activities taking place in a city over two days. We compared the model’s output versus input to quantify its overall value to the counter-IED mission. We exercised the model twice: forensically to identify insurgents after the IED explosions and prospectively to identify insurgents as early as possible beforehand. Although better sensors led to better intermediate results, they did not always l...

A Real-Time High Performance Computation Architecture for Multiple Moving Target Tracking Based on Wide-Area Motion Imagery via Cloud and Graphic Processing Units.

This paper presents the first attempt at combining Cloud with Graphic Processing Units (GPUs) in a complementary manner within the framework of a real-time high performance computation architecture for the application of detecting and tracking multiple moving targets based on Wide Area Motion Imagery (WAMI). More specifically, the GPU and Cloud Moving Target Tracking (GC-MTT) system applied a front-end web based server to perform the interaction with Hadoop and highly parallelized computation functions based on the Compute Unified Device Architecture (CUDA©). The introduced multiple moving target detection and tracking method can be extended to other applications such as pedestrian tracking, group tracking, and Patterns of Life (PoL) analysis. The cloud and GPUs based computing provides an efficient real-time target recognition and tracking approach as compared to methods when the work flow is applied using only central processing units (CPUs). The simultaneous tracking and recognition results demonstrate that a GC-MTT based approach provides drastically improved tracking with low frame rates over realistic conditions.

A method for real-time memory efficient implementation of blob detection in large images

In this paper we propose a method for real-time blob detection in large images with low memory cost. The method is suitable for implementation on the specialized parallel hardware such as multi-core platforms, FPGA and ASIC. It uses parallelism to speed-up the blob detection. The input image is divided into blocks of equal sizes to which the maximally stable extremal regions (MSER) blob detector is applied in parallel. We propose the usage of multiresolution analysis for detection of large blobs which are not detected by processing the small blocks. This method can find its place in many applications such as medical imaging, text recognition, as well as video surveillance or wide area motion imagery (WAMI). We explored the possibilities of usage of detected blobs in the feature-based image alignment as well. When large images are processed, our approach is 10 to over 20 times more memory efficient than the state of the art hardware implementation of the MSER.