Real-time Multimedia Processing Research Articles

A Study for Development of Digital Contents Management Systems Based on Smart Home

With the development of high-speed wireless LAN, the use of multimedia-type digital contents is increasing significantly even in the smart home environment. Therefore, it is very important to efficiently process and manage the use of digital content in the form of multimedia. In order to effectively process multimedia digital contents in the wireless LAN environment of a smart home, a server higher than the PC level is required. There is also a need for a support system that can exchange multimedia digital contents between clients of home appliances. Therefore, in this paper, we propose a Smart Home Digital Contents Management and Operation System (SDCMOS) that effectively processes and supports multimedia digital contents. This system is designed to effectively process real-time multimedia processing in home service in a wireless LAN environment. In addition, it was made possible to efficiently search multimedia information directly from the home server. SDCMOS is designed to be usable in fields such as biomedical image search, history museums, art exhibition halls, tourism information, geographic information, and e-commerce.

Salient Object Detection by Spatiotemporal and Semantic Features in Real-Time Video Processing Systems

Object detection is significant for event analysis in various intelligent multimedia processing systems. Although there have been many studies conducting research in this area, effective and efficient object detection methods for video sequences are still much desired. In this article, we investigate salient object detection in real-time multimedia processing systems. Considering the intrinsic relationship between top-down and bottom-up saliency features, we present a new effective method for video salient object detection based on deep semantic and spatiotemporal cues. After extracting top-down semantic features for object perception by a 2-D convolutional network, we concatenate them with bottom-up spatiotemporal cues for motion perception extracted by a 3-D convolutional network. In order to combine these features effectively, we feed them into a 3-D deconvolutional network for feature-sharing learning between semantic features and spatiotemporal cues for the final saliency prediction. Additionally, we propose a novel Gaussian-like loss function with an L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> -norm regularization term for parameter learning. Experimental results show that the proposed salient object detection approach performs better in terms of both effectiveness and efficiency for video sequences compared with the state-of-the-art models.

Real-time imaging-based assessment model for improving teaching performance and student experience in e-learning

Multimedia is an essential and integral part of electronic learning (e-learning). In this study, teaching performance and student learning experience are measured using real-time multimedia processing tools and techniques for the e-learning paradigm. Visual attention and visual engagement analysis are performed using two developed algorithms. Video lectures are recorded and delivered to students in e-learning pedagogical setup, which are examined for the visual attention and visual engagement of the student and teacher, respectively. Proposed methodology integrates the assessment on both student and teacher ends. Multimedia processing of video lectures for teaching performance produces scoring dataset. The same methodology on student end for visual attention is used to investigate student experience. These types of datasets then reduced to time-based datasets from the image-based dataset. Correlation and association of both datasets provide the opportunity to relate both student experience and teaching performance as well as to move forward to create content that is more useful. Computational performance of the developed algorithms is compared using different video lectures with their processed frames per second, which is analyzed as per their corresponding bins. Mean, max, and median of the processed frames of all the processed videos are also compared.

Energy-aware memory management for embedded multidimensional signal processing applications

In real-time data-intensive multimedia processing applications, data transfer and storage significantly influence, if not dominate, all the major cost parameters of the design space—namely energy consumption, performance, and chip area. This paper presents an electronic design automation (EDA) methodology for the high-level design of hierarchical memory architectures in embedded data-intensive applications, mainly in the area of multidimensional signal processing. Different from the previous works, the problems of data assignment to the memory layers, of mapping the signals into the physical memories, and of banking the on-chip memory are addressed in a consistent way, based on the same formal model. This memory management framework employs techniques specific to the integral polyhedra based dependence analysis. The main design target is the reduction of the static and dynamic energy consumption in the hierarchical memory subsystem.

Improving Energy Efficiency in FPGA Through Judicious Mapping of Computation to Embedded Memory Blocks

Field-programmable gate arrays (FPGAs) are being increasingly used as a preferred prototyping and accelerator platform for diverse application domains, such as digital signal processing (DSP), security, and real-time multimedia processing. However, mapping of these applications to FPGA typically suffers from poor energy efficiency because of high energy overhead of programmable interconnects (PI) in FPGA devices. This paper presents an energy-efficient heterogenous application mapping framework in FPGA, where the conventional application mappings to logic and DSP blocks (for DSP-enhanced FPGA devices) are combined with judicious mapping of specific computations to embedded memory blocks. A complete mapping methodology including functional decomposition, fusion, and optimal packing of operations is proposed and efficiently used to reduce the large energy overhead of PIs. Effectiveness of the proposed methodology is verified for a set of common applications using a commercial FPGA system. Experimental results show that the proposed heterogenous mapping approach achieves significant energy improvement for different input bit-widths (e.g., more than 35% of energy savings with 8 bit or smaller bit inputs compared to the corresponding mapping in configurable logic blocks). For further reduction of energy, we propose an energy/accuracy tradeoff approach, where the input operand bit-width is dynamically truncated to reduce memory area and energy at the expense of modest degradation in output-accuracy. We show that using a preferential truncation method, up to 88.6% energy savings can be achieved in a 32-tap finite impulse response filter with modest impact on the filter performance.

An Advanced MapReduce: Cloud MapReduce, Enhancements and Applications

Recently, Cloud Computing is attracting great attention due to its provision of configurable computing resources. MapReduce (MR) is a popular framework for data-intensive distributed computing of batch jobs. MapReduce suffers from the following drawbacks: 1. It is sequential in its processing of Map and Reduce Phases 2. Being cluster based, its scalability is relatively limited. 3. It does not support flexible pricing. 4. It does not support stream data processing. We describe Cloud MapReduce (CMR), which overcomes these limitations. Our results show that CMR is more efficient and runs faster than other implementations of the MR framework. In addition to this, we showcase how CMR can be further enhanced to: 1. Support stream data processing in addition to batch data by parallelizing the Map and Reduce phases through a pipelining model. 2. Support flexible pricing using Amazon Cloud's spot instances and to deal with massive machine terminations caused by spot price fluctuations. 3. Improve throughput and speed-up processing over traditional MR by more than 30% for large data sets. 4. Provide added flexibility and scalability by leveraging features of the cloud computing model. Click-stream analysis, real-time multimedia processing, time-sensitive analysis and other stream processing applications can also be supported.

Develop Monitoring System with Java Media Frame

This paper describes research in the use of The Java Media Framework (JMF) to develop Monitoring applications.. This paper introduces the structure of JMF library and describes the implementation of algorithms in our library. The Java Media Framework (JMF) is a recent API for Java dealing with real-time multimedia presentation and effects processing. JMF handles time-based media, media which changes with respect to time. Examples of this are video from a television source, audio from a raw-audio format file and animations.

Poly-DWT

Many modern computing applications have been enabled through the use of real-time multimedia processing. While several hardware architectures have been proposed in the research literature to support such primitives, these fail to address applications whose performance and resource requirements have a dynamic aspect. Embedded multimedia systems typically need a power and computation efficient design in addition to good compression performance. In this article, we introduce a Polymorphic Wavelet Architecture (Poly-DWT) as a crucial building block towards the development of embedded systems to address such challenges. We illustrate how our Poly-DWT architecture can potentially make dynamic resource allocation decisions, such as the internal bit representation and the processing kernel, according to the application requirements. We introduce a filter switching architecture that allows for dynamic switching between 5/3 and 9/7 wavelet filters and leads to a more power efficient design. Further, a multiplier-free design with a low adder requirement demonstrates the potential of Poly-DWT for embedded systems. Through an FPGA prototype, we perform a quantitative analysis of our Poly-DWT architecture, and compare our filter to existing approaches to illustrate the area and performance benefits inherent in our approach.

Optimality and Improvement of Dynamic Voltage Scaling Algorithms for Multimedia Applications

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> The high complexity and time-varying workload of emerging multimedia applications poses a major challenge for dynamic voltage scaling (DVS) algorithms. Although many DVS algorithms have been proposed for real-time applications, an efficient method for evaluating the optimality of such DVS algorithms for multimedia applications does not yet exist. In this paper, we propose the first offline linear programming (LP) method to determine the minimum energy consumption for processing multimedia tasks under stringent delay deadlines. On the basis of the obtained energy lower bound, we evaluate the optimality of various existing DVS algorithms. Furthermore, we extend the LP formulation in order to construct an online DVS algorithm for real-time multimedia processing based on robust sequential linear programming. Simulation results obtained by decoding a wide range of video sequences show that, on average, our online algorithm provides a scheduling solution that requires less than 0.3% more energy than the optimal lower bound with only 0.03% miss rate. In comparison, a very recent algorithm consumes approximately 4% more energy than the optimal lower bound at the same miss rate. </para>

Real-time multimedia processing in video sensor networks

Video sensor networks (VSNs) has become the recent research focus due to the rich information it provides to address various data-hungry applications. However, VSN implementations face stringent constraints of limited communication bandwidth, processing capability, and power supply. In-network processing has been proposed as efficient means to address these problems. The key component of in-network processing, task mapping and scheduling problem, is investigated in this paper. Although task mapping and scheduling in wired networks of processors has been extensively studied, their application to VSNs remains largely unexplored. Existing algorithms cannot be directly implemented in VSNs due to limited resource availability and shared wireless communication medium. In this work, an application-independent task mapping and scheduling solution in multi-hop VSNs is presented that provides real-time guarantees to process video feeds. The processed data is smaller in volume which further releases the burden on the end-to-end communication. Using a novel multi-hop channel model and a communication scheduling algorithm, computation tasks and associated communication events are scheduled simultaneously with a dynamic critical-path scheduling algorithm. Dynamic voltage scaling (DVS) mechanism is implemented to further optimize energy consumption. According to the simulation results, the proposed solution outperforms existing mechanisms in terms of guaranteeing application deadlines with minimum energy consumption.

Memory Size Computation for Real-Time Multimedia Applications Based on Polyhedral Decomposition

In real-time multimedia processing systems a very large part of the power consumption is due to the data storage and data transfer. Moreover, the area cost is often largely dominated by the memory modules. In deriving an optimized (for area and/or power) memory architecture, memory size computation is an important step in the exploration of the possible algorithmic specifications of multimedia applications. This paper presents a novel non-scalar approach for computing exactly the memory size in real-time multimedia algorithms. This methodology uses both algebraic techniques specific to the data-flow analysis used in modern compilers and, also, more recent advances in the theory of polyhedra. In contrast with all the previous works which are only estimation methods, this approach performs exact memory computations even for applications significantly large in terms of the code size, number of scalars, and number of array references.

Qualitative Study of Mobile Computing Oriented Architectures

The increase of the wireless networks has created a significant growth in demand for portable devices with different types of services such as text processing, connectivity to different networks, remote data base access and multimedia processing. In order to respond to these requirements, a number of different hardware architectures have been proposed recently, designed principally to integrate useful characteristics such as high performance for multimedia functions, energy efficient handling, small size and a low complexity. This work presents a general analysis of several architectures designed principally for real time multimedia processing in mobile applications. This article describes the advantages of these architectures and raises comparison strategy among them.

Software-controlled processor speed setting for low-power streaming multimedia

In this paper, we describe a software-controlled approach for adaptively minimizing energy in embedded systems for real-time multimedia processing. Energy is optimized by clock speed setting: the software controller dynamically adjusts processor clock speed to the frame rate (FR) requirements of the incoming multimedia stream. The speed-setting policy is based on a system model that correlates clock speed with best case, average case, and worst case sustainable FRs, accounting for data dependency in multimedia streams. The technique has been implemented in a energy-efficient MPEG3 real-time decoder algorithm designed for wearable devices as a case study. The target system is the Hewlett-Packard SmartBadgeIII prototype system based on the StrongARM1100 processor. Hardware measurements show that computational energy can be drastically reduced (up to 40%) with respect to fixed-frequency operation.

KAIST image computing system (KICS): A parallel architecture for real-time multimedia data processing

An efficient parallel architecture is proposed for high-performance multimedia data processing using multiple multimedia video processors (MVP; TMS320C80), which are fully programmable general digital signal processors (DSP). This paper describes several requirements for a multimedia data processing system and the system architecture of an image computing system called the KAIST Image Computing System (KICS). The performance of the KICS is evaluated in terms of its I/O bandwidth and the execution time for some image processing functions. An application of the KICS to the real-time Moving Picture Expert Group 2 (MPEG-2) encoder is introduced. The programmability and the high-speed data-access capability of the KICS are its most important features as a high-performance system for real-time multimedia data processing.