Image Stream Research Articles

Object detection, recognition, and tracking from UAVs using a thermal camera

AbstractIn this paper a multiple object detection, recognition, and tracking system for unmanned aerial vehicles (UAVs) has been studied. The system can be implemented on any UAVs platform, with the main requirement being that the UAV has a suitable onboard computational unit and a camera. It is intended to be used in a maritime object tracking system framework for UAVs, which enables a UAV to perform multiobject tracking and situational awareness of the sea surface, in real time, during a UAV operation. Using machine vision to automatically detect objects in the camera's image stream combined with the UAV's navigation data, the onboard computer is able to georeference each object detection to measure the location of the detected objects in a local North‐East (NE) coordinate frame. A tracking algorithm which uses a Kalman filter and a constant velocity motion model utilizes an object's position measurements, automatically found using the object detection algorithm, to track and estimate an object's position and velocity. Furthermore, a global‐nearest‐neighbor algorithm is applied for data association. This is achieved using a measure of distance that is based not only on the physical distance between an object's estimated position and the measured position, but also how similar the objects appear in the camera image. Four field tests were conducted at sea to verify the object detection and tracking system. One of the flight tests was a two‐object tracking scenario, which is also used in three scenarios with an additional two simulated objects. The tracking results demonstrate the effectiveness of using visual recognition for data association to avoid interchanging the two estimated object trajectories. Furthermore, real‐time computations performed on the gathered data show that the system is able to automatically detect and track the position and velocity of a boat. Given that the system had at least 100 georeferenced measurements of the boat's position, the position was estimated and tracked with an accuracy of 5–15 m from 400 m altitude while the boat was in the camera's field of view (FOV). The estimated speed and course would also converge to the object's true trajectories (measured by Global Positioning System, GPS) for the tested scenarios. This enables the system to track boats while they are outside the FOV of the camera for extended periods of time, with tracking results showing a drift in the boat's position estimate down to 1–5 m/min outside of the FOV of the camera.

Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio

The rhesus macaque is an important model species in several branches of science, including neuroscience, psychology, ethology, and medicine. The utility of the macaque model would be greatly enhanced by the ability to precisely measure behavior in freely moving conditions. Existing approaches do not provide sufficient tracking. Here, we describe OpenMonkeyStudio, a deep learning-based markerless motion capture system for estimating 3D pose in freely moving macaques in large unconstrained environments. Our system makes use of 62 machine vision cameras that encircle an open 2.45 m × 2.45 m × 2.75 m enclosure. The resulting multiview image streams allow for data augmentation via 3D-reconstruction of annotated images to train a robust view-invariant deep neural network. This view invariance represents an important advance over previous markerless 2D tracking approaches, and allows fully automatic pose inference on unconstrained natural motion. We show that OpenMonkeyStudio can be used to accurately recognize actions and track social interactions.

A Stream Algebra for Performance Optimization of Large Scale Computer Vision Pipelines.

There is a large growth in hardware and software systems capable of producing vast amounts of image and video data. These systems are rich sources of continuous image and video streams. This motivates researchers to build scalable computer vision systems that utilize data-streaming concepts for processing of visual data streams. However, several challenges exist in building large-scale computer vision systems. For example, computer vision algorithms have different accuracy and speed profiles depending on the content, type, and speed of incoming data. Also, it is not clear how to adaptively tune these algorithms in large-scale systems. These challenges exist because we lack formal frameworks for building and optimizing large-scale visual processing. This paper presents formal methods and algorithms that aim to overcome these challenges and improve building and optimizing large-scale computer vision systems. We describe a formal algebra framework for the mathematical description of computer vision pipelines for processing image and video streams. The algebra naturally describes feedback control and provides a formal and abstract method for optimizing computer vision pipelines. We then show that a general optimizer can be used with the feedback-control mechanisms of our stream algebra to provide a common online parameter optimization method for computer vision pipelines.

Contextual and Spatial Associations Between Objects Interactively Modulate Visual Processing.

Much of what we know about object recognition arises from the study of isolated objects. In the real world, however, we commonly encounter groups of contextually associated objects (e.g., teacup and saucer), often in stereotypical spatial configurations (e.g., teacup above saucer). Here we used electroencephalography to test whether identity-based associations between objects (e.g., teacup–saucer vs. teacup–stapler) are encoded jointly with their typical relative positioning (e.g., teacup above saucer vs. below saucer). Observers viewed a 2.5-Hz image stream of contextually associated object pairs intermixed with nonassociated pairs as every fourth image. The differential response to nonassociated pairs (measurable at 0.625 Hz in 28/37 participants) served as an index of contextual integration, reflecting the association of object identities in each pair. Over right occipitotemporal sites, this signal was larger for typically positioned object streams, indicating that spatial configuration facilitated the extraction of the objects’ contextual association. This high-level influence of spatial configuration on object identity integration arose ~ 320 ms post-stimulus onset, with lower-level perceptual grouping (shared with inverted displays) present at ~ 130 ms. These results demonstrate that contextual and spatial associations between objects interactively influence object processing. We interpret these findings as reflecting the high-level perceptual grouping of objects that frequently co-occur in highly stereotyped relative positions.

Fast Periodic Visual Stimulation indexes preserved semantic memory in healthy ageing

Behavioural studies investigating the preservation of semantic memory in healthy ageing have reported mixed findings. One suggested reason for this discrepancy is that the processes underpinning lexical access to semantic knowledge may be sensitive to ageing. It is therefore necessary to assess semantic memory utilising tasks that are not explicitly linguistic. In this study, a fast periodic visual stimulation (FPVS) paradigm coupled with EEG was used to assess the ability of younger and older adults to automatically distinguish between images by their semantic category. Participants were presented with a 6 Hz stream of images drawn from one semantic category except every fifth image (occurring at a rate of 1.2 Hz) which was drawn from an alternate semantic category. For both younger and older adults, results demonstrate successful and comparable semantic categorisation. This was detectable at the individual level for 71% and 72% of older and younger adults, respectively. Given the rapid presentation rate and absence of explicit instruction to categorise images, the task is unlikely to utilise linguistic strategies and suggests the maintenance of semantic memory in healthy ageing. Moreover, this study utilised mobile EEG equipment and short presentation times that would be suitable for practical application outside a research setting.

Statistical Learning Signals for Complex Visual Images in Macaque Early Visual Cortex.

Animals of several species, including primates, learn the statistical regularities of their environment. In particular, they learn the temporal regularities that occur in streams of visual images. Previous human neuroimaging studies reported discrepant effects of such statistical learning, ranging from stronger occipito-temporal activations for sequences in which image order was fixed, compared with sequences of randomly ordered images, to weaker activations for fixed-order sequences compared with sequences that violated the learned order. Several single-unit studies in macaque monkeys reported that after statistical learning of temporal regularities, inferior temporal (IT) neurons show reduced responses to learned fixed-order sequences of visual images compared with random or mispredicted sequences. However, it is unknown how other macaque brain areas respond to such temporal statistical regularities. To address this gap, we exposed rhesus monkeys (Macaca mulatta) to two types of sequences of complex images. The “regular” sequences consisted of a continuous stream of quartets, and within each quartet, the image order was fixed. The quartets themselves were displayed, uninterrupted, in a random order. The same monkeys were exposed to sequences of other images having a pseudorandomized order (“random” sequence). After exposure, both monkeys were scanned with functional MRI (fMRI) using a block design with three conditions: regular sequence, random sequence, and fixation-only blocks. A whole-brain analysis showed a reduced activation in mainly the occipito-temporal cortex for the regular compared to the random sequences. Marked response reductions for the regular sequence were observed in early extrastriate visual cortical areas, including area V2, despite the use of rather complex images of animals. These data suggest that statistical learning signals are already present in early visual areas of monkeys, even for complex visual images. These monkey fMRI data are in line with recent human fMRI studies that showed a reduced activation in early visual areas for predicted compared with mispredicted complex images.

Google’s lens: computational photography and platform capitalism

Computational photography is currently altering the representational and social functions of photographic imaging. A range of heavily automated computational processing techniques produce images that remediate digital photography to circumvent physical limitations associated with the size of smartphones, emulating the aesthetics associated with larger format digital cameras and professional photographic workflows and practices. These processes include automated compositing where images seen by users are constituted of up to 15 individual frames, the simulation of a shallow depth-of-field, automated facial retouching and even providing automated assistance to suggest alternative frames within the image stream to serve as the base image. This article explores these emerging techniques and accompanying claims that such processes are radically transforming photographic practice. While the extent and modes of automation and algorithmic processing depart from prior practices, contextualising them within the histories of photographic compositing and the algorithmic malleability of digital photography suggests the intensification of existing trends rather than an epistemic break. Furthermore, exploring the representational politics of automated facial retouching and the datafication of images situates these changes within the broader social context of dataveillance and platform capitalism.

Real-time wavelet transform for infinite image strips

This article presents a single-loop approach to a 2-D discrete wavelet transform that allows processing infinitely high-image strip-maps. The paper gradually compares several computational strategies to finally show how to deal with a multi-scale wavelet transform of infinite image streams. Besides, the transform is followed by a bit-plane encoder which also processes data in a single loop. The whole machinery is part of a CCSDS 122.0 image codec which manages to process a single pixel in about 33 ns on a contemporary desktop computer, without the contribution of any parallel computing or SIMD vectorization.

Categorization of objects and faces in the infant brain and its sensitivity to maternal odor: further evidence for the role of intersensory congruency in perceptual development

According to recent evidence, rapid categorization of natural face images in the infant brain is enhanced by concomitant maternal odor (Leleu et al., 2020). To test whether this effect is selective to faces, we recorded scalp electroencephalogram (EEG) in 4-month-old infants presented with variable exemplars of a nonface visual category - cars - appearing every 6 stimuli in 6-Hz streams of natural object images. At the same time, infants were exposed to the maternal or to a control odor context. A relatively weak neural categorization response to cars (i.e., a differential response to cars that generalizes across exemplars) was observed at 1 Hz over the right occipital cortex in both odor conditions, revealing rapid categorization of an unfamiliar object in the infant brain. However, the car categorization response was not modulated by maternal odor, suggesting that odors selectively prime neural activity in the infant visual cortex to categorize congruent incoming inputs.

Real-time 3D unstructured environment reconstruction utilizing VR and Kinect-based immersive teleoperation for agricultural field robots

Moving and operating autonomously in a field or orchard environment is challenging for agricultural robots due to the complex task requirements and highly unstructured conditions. The human–computer interaction-based remote control can provide the robots with an alternative solution to assist decision making and motion planning. In this study, a virtual reality (VR) and Kinect-based immersive teleoperation system were proposed to connect the physical and virtual world by utilizing real-time large-scale unstructured agricultural environment reconstruction and simultaneous virtual environment creation. The proposed system, with a relatively large server, can convert the scene into a realistic model by combing the depth and color image streams received from Kinect, and project them back into 3D (three dimensions) space in such a manner that the real 3D scene inside the camera’s field of view is recreated virtually. To create a VR environment for a VR headset, an optimized Bundlefusion-based algorithm was developed for real-time 3D reconstruction of the unstructured agricultural scene in the natural environment. Additionally, the performance of the proposed real-time 3D reconstruction algorithm was evaluated and compared with Bundlefusion and voxel hashing algorithms in different large-scale unstructured agricultural environments. Performing optimizing pose on our optimized algorithm leverages a large number of processing cores available to minimize the delay between data capture and rendering, and it reduces the average acquired time to process each frame no more than 0.9 ms. The experimental results including, lesscomputer storage occupied, fast frame processing time, and high-quality 3D realistic model indicate that our proposed system and algorithm have the potential applicability of immersive teleoperation in an unstructured agricultural environment.

Enabling Image-Based Streamflow Monitoring at the Edge

Monitoring streamflow velocity is of paramount importance for water resources management and in engineering practice. To this aim, image-based approaches have proved to be reliable systems to non-intrusively monitor water bodies in remote places at variable flow regimes. Nonetheless, to tackle their computational and energy requirements, offload processing and high-speed internet connections in the monitored environments, which are often difficult to access, is mandatory hence limiting the effective deployment of such techniques in several relevant circumstances. In this paper, we advance and simplify streamflow velocity monitoring by directly processing the image stream in situ with a low-power embedded system. By leveraging its standard parallel processing capability and exploiting functional simplifications, we achieve an accuracy comparable to state-of-the-art algorithms that typically require expensive computing devices and infrastructures. The advantage of monitoring streamflow velocity in situ with a lightweight and cost-effective embedded processing device is threefold. First, it circumvents the need for wideband internet connections, which are expensive and impractical in remote environments. Second, it massively reduces the overall energy consumption, bandwidth and deployment cost. Third, when monitoring more than one river section, processing “at the very edge” of the system efficiency improves scalability by a large margin, compared to offload solutions based on remote or cloud processing. Therefore, enabling streamflow velocity monitoring in situ with low-cost embedded devices would foster the widespread diffusion of gauge cameras even in developing countries where appropriate infrastructure might be not available or too expensive.

Efficient mining of IoT based data streams for advanced computer vision systems

With emergence of Internet of Things (IoT) and subsequent technologies, smart devices are being increasingly used in various domains such as smart homes, smart parking, intelligent transportation etc. Vast amount of image and video data has been produced by IoT based systems in the form of continuous and possibly infinite image and video streams. This demands the development of advanced predictive vision systems which exploits stream mining concepts for intelligent processing of visual data streams. Among other challenges faced by visual data streams, a major challenge is concept drift, which is caused by overtime change in data distribution. In the presence of skewed data, the detection of concept drift becomes more challenging. When analyzing the data generated from smart devices and other performance critical wireless sensors, concept drift affects data integrity and accuracy of prediction results. EWMA for Concept Drift Detection (ECDD) has been proposed in the literature for detecting data streams. However, ECDD has a high prediction error rate which makes it less useful for performance critical data streams generated by imaging and video data streams. In this paper, Vision based Drift Detection Method (VisDDM) is proposed, which systematically handles abrupt and gradual concept drift in data streams. Experiments have been performed using synthetic and real world datasets from different application domains. Our proposed VisDDM algorithm is able to handle abrupt and gradual drift types and outperformed the existing drift detection methods in terms of accuracy and mean evaluation time.

Video encryption based on hyperchaotic system

With the increasing popularity of multimedia data on the internet and our daily life, the way people get information is becoming more diverse, video is becoming an important part of our dailylife and work. At the same time, the popularity of video also brings new security risks and challenges, more and more people are paying attention to video security issues. Based on hyperchaotic system, this paper proposes and implements a video encryption scheme. The encryption algorithm divide a video stream into a color image stream and an audio stream. Color image is composed of three-dimensional channels of Y, Cb, and Cr, the amount of practical information carried by three channels is different. Therefore, an Anorld transforms and a DNA encoding algorithm is used for the Y channel with more information. The Cb and Cr channels with less information are encrypted by Lorenz hyperchaotic map. Audio information is encrypted by Logistic chaotic map. The paper thoroughly tested the feasibility of the process of encryption and decryption, analyzed the resistance of statistical attacks and completed tests on the histogram, information entropy, and correlation of neighboring pixels. The paper also tested the scheme’s ability to resist violent attacks and completed the analysis of key space, key sensitivity, signal-to-noise ratio, spectrogram, variance, and mean. At the end of the article we test the comparison of capability and efficiency of our scheme and other two methods. Results of experiment show that our scheme has better security and encryption efficiency.

Closed-Loop Software Architecture for Spacecraft Optical Navigation and Control Development

A software architecture is discussed to develop, run, and test novel autonomous visual spacecraft navigation and control methods in a realistic simulation. This architecture harnesses two main components: a high-fidelity, faster-than-real-time, astrodynamics simulation framework; and a sister software package to dynamically visualize the simulation environment. Maneuvers such as fly-bys and orbit insertions occur over short periods of time and must occur autonomously. Yet, there are no open-source software packages that provide fully coupled spacecraft environments and Flight Software (FSW) enabling Optical Navigation (OpNav) mission scenarios. The presented tool consists of the Basilisk∗ astrodynamics framework interfacing with a Unity-based visualization Vizard that provides a synthetic image stream of a camera sensor. This modular and extensible setup allows optical guidance, navigation and control (GNC) algorithms to be run in a closed-loop format purely in software. The optical measurements are generated in the visualization and passed to the simulation, allowing for real-time control and decision making. This Vizard software has the ability to import shape-models, planet maps, and move into an instrument point-of-view. Paired with open-source image processing libraries, these combined components provide all the necessary pieces to fully simulate autonomous, closed-loop, OpNav scenarios in a faster-than-real-time configuration. This allows for progress in the autonomy sector, as full-fledged FSW can be tested in a real flight environment. Furthermore, this enables more realistic and extensive testing of the software, which in turn increases reliability of the GNC methods as they are refined. This paper presents the Basilisk and Vizard interface architecture, its performance, and develops a example scenario. The image processing methods are displayed and the visualization scenes are validated for pointing purposes, which in turns allows to develop an autonomous pointing algorithm developed in this software environment.

Estimation of boundary conditions for patient-specific liver simulation during augmented surgery

Augmented reality can improve the outcome of hepatic surgeries, assuming an accurate liver model is available to estimate the position of internal structures. While researchers have proposed patient-specific liver simulations, very few have addressed the question of boundary conditions. Resulting mainly from ligaments attached to the liver, they are not visible in preoperative images, yet play a key role in the computation of the deformation. We propose to estimate both the location and stiffness of ligaments by using a combination of a statistical atlas, numerical simulation, and Bayesian inference. Ligaments are modeled as polynomial springs connected to a liver finite element model. They are initialized using an anatomical atlas and stiffness properties taken from the literature. These characteristics are then corrected using a reduced-order unscented Kalman filter based on observations taken from the laparoscopic image stream. Our approach is evaluated using synthetic data and phantom data. By relying on a simplified representation of the ligaments to speed up computation times, it is not estimating the true characteristics of ligaments. However, results show that our estimation of the boundary conditions still improves the accuracy of the simulation by 75% when compared to typical methods involving Dirichlet boundary conditions. By estimating patient-specific boundary conditions, using tracked liver motion from RGB-D data, our approach significantly improves the accuracy of the liver model. The method inherently handles noisy observations, a substantial feature in the context of augmented reality.

Radiology’s Information Architecture Could Migrate to One Emulating That of Smartphones

Diagnostic radiology (DxR), having had successful serial co-evolutions with imaging equipment and PACS, is faced with another. With a backdrop termed "globotics transition," it should create an IT and informatics infrastructure capable of integrating artificial intelligence (AI) into current critical communication functions of PACS and incorporating functions currently residing in balkanized products. DxR will face the challenge of adopting sustaining and disruptive AI innovations simultaneously. In this co-evolution, a major selection force for AI will be increasing the flow of information and patients; "increasing" means faster flow over larger areas defined by geography and content. Larger content includes a broader spectrum of imaging and nonimaging information streams that facilitate medical decision making. Evolution to faster flow will gravitate toward a hierarchical IT architecture consisting of many small channels feeding into fewer larger channels, something potentially difficult for current PACS. Smartphone-like architecture optimized for communication and integration could provide a large-channel backbone and many smaller feeding channels for basic functions, as well as those needing to innovate rapidly. New, more flexible architectures stimulate market competition in which DxR could act as an artificial selection force to influence development of faster increased flow in current PACS companies, in disruptors such as consolidated AI companies, or in entirely new entrants like Apple or Google. In this co-evolution, DxR should be able to stimulate design of a modern communication medium that increases the flow of information and decreases the time and energy necessary to absorb it, thereby creating even more indispensable clinical value for itself.

Affective Bias without Hemispheric Competition: Evidence for Independent Processing Resources in Each Cortical Hemisphere

We assessed the extent of neural competition for attentional processing resources in early visual cortex between foveally presented task stimuli and peripheral emotional distracter images. Task-relevant and distracting stimuli were shown in rapid serial visual presentation (RSVP) streams to elicit the steady-state visual evoked potential, which serves as an electrophysiological marker of attentional resource allocation in early visual cortex. A task-related RSVP stream of symbolic letters was presented centrally at 15 Hz while distracting RSVP streams were displayed at 4 or 6 Hz in the left and right visual hemifields. These image streams always had neutral content in one visual field and would unpredictably switch from neutral to unpleasant content in the opposite visual field. We found that the steady-state visual evoked potential amplitude was consistently modulated as a function of change in emotional valence in peripheral RSVPs, indicating sensory gain in response to distracting affective content. Importantly, the facilitated processing for emotional content shown in one visual hemifield was not paralleled by any perceptual costs in response to the task-related processing in the center or the neutral image stream in the other visual hemifield. Together, our data provide further evidence for sustained sensory facilitation in favor of emotional distracters. Furthermore, these results are in line with previous reports of a "different hemifield advantage" with low-level visual stimuli and are suggestive of independent processing resources in each cortical hemisphere that operate beyond low-level visual cues, that is, with complex images that impact early stages of visual processing via reentrant feedback loops from higher order processing areas.

Investigation of nanostar-labeled mesenchymal stem cells for in vivo cell tracking in osteoarthritis using optoacoustic imaging

Background & Aim This project aims to investigate gold nanostar (NS)-labeled mesenchymal stem/stromal cells (MSCs) for in vivo cell tracking using optoacoustic imaging (OAI) for the treatment of osteoarthritis (OA). OAI, specifically multi-spectral optoacoustic tomography (MSOT) has demonstrated excellent resolution at increasing depths. To further enhance depth penetration and sensitivity, a novel gold nanoparticle – the NS has been developed. This NS will also be used to permit tracking of MSCs in vivo. Different nanostar surface coatings have been investigated for their uptake and effects on MSC characteristics: methoxy (NS-OMe) and carboxyl (NS-COOH). SPIONs (superparamagnetic iron oxide nanoparticles) have also been used to functionalize the NS (NS-SPIONs) and permit dual OAI and magnetic resonance imaging. In order to permit labeling and tracking of MSC-derived extracellular vesicles (EVs), conjugation of an antibody to the NS has also been performed using available carboxyl groups. The aims of this project are to optimize MSC labeling and to investigate MSC phenotype post-labeling. Methods, Results & Conclusion Methods Methods to investigate uptake of NS included transmission electron microscopy (TEM), Image Stream X analysis (ISX) and MSOT. Viability, tri-lineage differentiation and surface immunophenotype assays were used to assess MSC characteristics post -labeling. In vivo imaging of NS in the knee joint was performed using MSOT. Results NS-OMe were efficiently internalized by MSCs and had no effects on MSCs characteristics. NS-COOH demonstrated inefficient cellular internalization. ISX was used to quantify the uptake of NS conjugated to an antibody and validated these initial findings as only 20% of MSCs contained NS-COOH conjugated to an antibody. NS-SPIONs were also investigated and demonstrated efficient internalization by MSCs. NS-SPIONS will be further investigated to identify the optimal labeling concentration for MSOT detection. Finally, preliminary work has explored the capacity of MSOT to detect NS in vivo. An enhanced optoacoustic signal in the knee joint following an intra-articular injection of NS was shown as indicated by the white arrows in figure 1. Conclusion MSC uptake and internalization of NS is variable in response to different NS surface coatings. NS-OMe demonstrated the most efficient uptake by cells while NS-COOH was least efficient. The uptake of NS-SPIONs has been found to be optimal and future work will determine the optimal concentration for MSOT detection.

Frequency-Tagging Electroencephalography of Superimposed Social and Non-Social Visual Stimulation Streams Reveals Reduced Saliency of Faces in Autism Spectrum Disorder.

Individuals with autism spectrum disorder (ASD) have difficulties with social communication and interaction. The social motivation hypothesis states that a reduced interest in social stimuli may partly underlie these difficulties. Thus far, however, it has been challenging to quantify individual differences in social orientation and interest, and to pinpoint the neural underpinnings of it. In this study, we tested the neural sensitivity for social versus non-social information in 21 boys with ASD (8-12 years old) and 21 typically developing (TD) control boys, matched for age and IQ, while children were engaged in an orthogonal task. We recorded electroencephalography (EEG) during fast periodic visual stimulation (FPVS) of social versus non-social stimuli to obtain an objective implicit neural measure of relative social bias. Streams of variable images of faces and houses were superimposed, and each stream of stimuli was tagged with a particular presentation rate (i.e., 6 and 7.5 Hz or vice versa). This frequency-tagging method allows disentangling the respective neural responses evoked by the different streams of stimuli. Moreover, by using superimposed stimuli, we controlled for possible effects of preferential looking, spatial attention, and disengagement. Based on four trials of 60 s, we observed a significant three-way interaction. In the control group, the frequency-tagged neural responses to faces were larger than those to houses, especially in lateral occipito-temporal channels, while the responses to houses were larger over medial occipital channels. In the ASD group, however, faces and houses did not elicit significantly different neural responses in any of the regions. Given the short recording time of the frequency-tagging paradigm with multiple simultaneous inputs and the robustness of the individual responses, the method could be used as a sensitive marker of social preference in a wide range of populations, including younger and challenging populations.

Automatic Transformation of a Video Using Multimodal Information for an Engaging Exploration Experience

Exploring the content of a video is typically inefficient due to the linear streamed nature of its media and the lack of interactivity. While different approaches have been proposed for enhancing the exploration experience of video content, the general view of video content has remained basically the same, that is, a continuous stream of images. It is our contention that such a conservative view on video limits its potential value as a content source. This paper presents An Alternative Representation of Video via feature Extraction (RAAVE), a novel approach to transform videos from a linear stream of content into an adaptive interactive multimedia document and thereby enhance the exploration potential of video content by providing a more engaging user experience. We explore the idea of viewing video as a diverse multimedia content source, opening new opportunities and applications to explore and consume video content. A modular framework and algorithm for the representation engine and template collection is described. The representation engine based approach is evaluated through development of a prototype system grounded on the design of the proposed approach, allowing users to perform multiple content exploration tasks within a video. The evaluation demonstrated RAAVE’s ability to provide users with a more engaging, efficient and effective experience than a typical multimedia player while performing video exploration tasks.