Heterogeneous Multimedia Research Articles

Algorithms for Face Image Mutual Reconstruction by Means of Two-Dimensional Projection Methods

In this paper, we consider the problem of mutual reconstruction of face image pairs. We addressed this problem in our previous article, where the proposed solutions were discussed in connection with Heterogeneous Face Recognition and Cross-Modal Multimedia Retrieval problems. Those solutions are based on one-dimensional and two-dimensional Principal Component Analysis performed over two original face images followed by their projection on independent eigenspaces, estimation of a transformation matrix and mutual reconstruction of the face image by means of one-dimensional and two-dimensional Karhunen-Loeve Transform. In this article, we propose new approaches and solutions, which are based solely on the two-dimensional eigenspace projection methods, and two regression models — Multiple Linear Regression and Partial Least Squares regression. We present the experiments on mutual reconstruction of face images in sketch/photo pairs, in pairs of face images with age-related changes, and in pairs of 2D/3D face images. In order to conduct the experiments, we selected two variants of the proposed approach. First one is based on two-dimensional Principal Component Analysis and Partial Least Squares regression, and the second one is based on two-dimensional Partial Least Squares and Multiple Linear Regression. Both variants showed acceptable performance for practical applications involving the mutual reconstruction of face images. Furthermore, we consider the method to improve the quality of reconstructed face images in the case of mixed datasets. This method involves classification of the dataset by means of two-dimensional Linear Discriminant Analysis and fitting of a separate regression model for each class. In addition, we show that generally, mutual reconstruction of face images is also achievable in conditions when original images are not a part of training sets of face images.

Summary Extract Approach of Heterogeneous Multimedia Documents Based on Folksonomy with Incentive and Quality Assurance Mechanisms

Summary extract of heterogeneous multimedia documents is an important content and requirement of information management at the Internet era. Though some summary extract approaches have been proposed in recent years, in general they can only be used in a single-type media. More important is that the existed approaches of multimedia summary extract had no satisfactory effects, and can’t be brought into practice. Aiming at these problems, an approach based on folksonomy with incentive and quality assurance mechanisms is put forward. The proven folksonomy ensures the practicality of the approach, the incentive and quality assurance mechanisms avoid the innate problems of folksonomy: cold boot and inaccurate tagging. The experiment shows that the approach is reasonable and effective.

Effective Software Installation for Embedded Software by Applying the Reverse Engineering Approach

Background/Objectives: Reverse Engineering of Electronic Programme Resources (EPR) is the process of reading the binary data from the files having “.bin” extension and presented in a well-structured format. Method/Analysis: The resources used with the EP resources are text, pictures, colors and UI controls, each of these are stored in XML format then converted to .bin extension files to be flashed on an embedded device. If the developer wishes to see the contents, it is very difficult to analyze the format as well as the data, as it is in binary format and needs an expert to interpret it. Findings: The required concept can be shown visually by which developer will know the format readily, and this work tries to satisfy the requirement of analyzing the binary format for heterogeneous multimedia embedded software systems. Applications/Improvements: The research introduced can be applied to any software binaries that are to be flashed on an embedded device obtaining the necessary binary format.

A Joint CAC and Dynamic Bandwidth Allocation Technique for Capacity and QoS Analysis in Heterogeneous LTE based BWA Network: Few Case Studies

Broadband Wireless Access (BWA) network such as LTE brings heterogeneity in terms of different radio access technologies, multi-tier architectures and co-implementation of various protocols to support various types of services with desired Quality of Service (QoS) for the subscribers. However, due to limited bandwidth and high demand of LTE-BWA networks, effective network planning is required to utilize the available radio resources in an efficient way. To achieve this target, this paper aims to implement a unique Radio Resource Management scheme for a heterogeneous LTE based BWA network (BWA Het-Nets) which consists of Small cells along with Macro cells. The proposed analysis has two components: (1) Received Signal Strength (RSS) based CAC policy; and (2) Dynamic Bandwidth Allocation (DBA) scheme for QoS provisioning. The novelty of this work lies on the integration of the RSS based CAC policy with DBA scheme for improved performance in terms of different QoS parameters like New Call Blocking Probability (NCBP), Hand off Call Dropping Probability (HCDP) and Bandwidth Utilization (BU). A Continuous Time Markov Chain based mathematical model is also developed for the realistic analysis of those QoS parameters. The Joint CAC and DBA scheme (JCAC-DBA) in two-tier LTE BWA Het-Nets significantly improves NCBP, HCDP, BU and overall capacity of the network under different combinations of macro/small cells for heterogeneous multimedia services.

SOR: An optimized semantic ontology retrieval algorithm for heterogeneous multimedia big data

Semantic information can express the search intentions of users, and this approach has become an important tool in the field of information retrieval. To support semantic-based multimedia retrieval in big data environment, this paper presents an optimized algorithm called semantic ontology retrieval (SOR), which uses big data processing tools to store and retrieve ontologies from heterogeneous multimedia data. First, the background of semantic extraction and ontology representation for multimedia big data are addressed. Second, the methodology of SOR, including the model definition and retrieval algorithm, is proposed. Third, for parallel processing SOR in distributed nodes, a MapReduce-based retrieval framework is presented. Finally, to achieve high retrieval precision and good user experience, a user feedback scheme is designed. The experimental results illustrate that SOR is suitable for semantic-based retrieval for heterogeneous multimedia big data.

Guest editorial: web multimedia semantic inference using multi-cues

With the popularity of social media applications and Web 2.0 techniques, user-generated multimedia contents such as blogs, test messages, photos, videos, user click log and Place of Interest (POI) check-ins become pervasive, which enables the study on exploiting them as multiple cues for web multimedia semantic inference. Most of the time when one speaks of web multimedia corpora, he/she may think of heterogeneous corpora consisting of data from various sources, and of different modality. The heterogeneous multimedia content provides a variety of cues for semantic inference of real-world multimedia applications. Research so far has mostly focused on mono-cue analysis of multimedia content, such as looking only into images, videos, or text, but rarely leverage multiple semantic cues like the surrounding texts of images/videos on a web page or the click logs of users’ profiles from the same community. As such, new algorithms and models for analyzing correlations among multiple semantic cues become one of the most active research areas in web multimedia applications. From the above background, many efforts have focused on the utilization of multiple semantic cues for web multimedia semantic inference. Particularly, the different semantic cues may be temporally synchronized (e.g., video clips and corresponding audio transcripts, animations, multimedia presentations), spatially related (images embedded in text, object relationships in 3D space), semantically correlated (combined analysis of collections of videos, set of images created by one’s social network), or otherwise click-through connected (images World Wide Web (2016) 19:177–179 DOI 10.1007/s11280-015-0360-2

Internet of multimedia things: Vision and challenges

Internet of Things (IoT) systems cannot successfully realize the notion of ubiquitous connectivity of everything if they are not capable to truly include ‘multimedia things’. However, the current research and development activities in the field do not mandate the features of multimedia objects, thus leaving a gap to benefit from multimedia content based services and applications. In this paper, we analyze this issue by contemplating the concept of IoT and drawing an inspiration towards the perspective vision of ‘Internet of Multimedia Things’ (IoMT). Therein, we introduce IoMT as a novel paradigm in which smart heterogeneous multimedia things can interact and cooperate with one another and with other things connected to the Internet to facilitate multimedia based services and applications that are globally available to the users. Some applications and use-cases for IoMT are presented to reflect the possibilities enabled by this new paradigm. An IoMT architecture is then presented which is segregated into four distinct stages; (i) multimedia sensing, (ii) reporting and addressability, (iii) multimedia-aware cloud, and (iv) multi-agent systems. Instead of proposing specific technical solutions for each individual stage of the presented architecture, we survey the already existing technologies, providing a synthesis for the realization of the vision of IoMT. Subsequently, various requirements and challenges as well as the feasibility of existing solutions for each stage of proposed IoMT architecture are comprehensively discussed.

Scalable Video Multicasting: A Stochastic Game Approach With Optimal Pricing

Heterogeneous multimedia content delivery over wireless networks is an important yet challenging issue. One of the challenges is maintaining the quality of service due to scarce resources in wireless communications and heavy loadings from heterogeneous demands. A promising solution is combining multicasting and scalable video coding (SVC) techniques via cross-layer design, which has been shown to effectively enhance the quality of multimedia content delivery service in the literature. Nevertheless, most existing works on SVC multicasting system focus on the static scenarios, where a snapshot of user demands is given and remains the same. In addition, the economic value of the SVC multicasting system, which is an important issue from the service provider's perspective, has seldom been explored. In this paper, we study a subscription-based SVC multicasting system with stochastic user arrival and heterogeneous user preferences. A stochastic framework based on the multidimensional Markov decision process (M-MDP) is proposed to study the negative network externality existing in the proposed system and theoretically evaluate the corresponding system efficiency. A game-theoretic analysis is conducted to understand the rational demands from heterogeneous users under different subscription pricing schemes. By transforming the original dynamic and complex M-MDP revenue optimization problem into a traditional average-reward MDP problem, we show that the optimal pricing strategy that maximizes the expected revenue of the service provider can be derived efficiently. Moreover, the overall user's valuation on the system, e.g., social welfare, is maximized under such an optimal pricing strategy. Finally, the efficiency of the proposed solutions is evaluated through simulations.

An effective and economical architecture for semantic-based heterogeneous multimedia big data retrieval

Data variety has been one of the most critical features for multimedia big data. Some multimedia documents, although in different data formats and storage structures, often express similar semantic information. Therefore, the way to manage and retrieve multimedia documents reflecting users’ intent in heterogeneous big data environments has become an important issue. In this paper, we present an effective and economical architecture named SHMR (Semantic-based Heterogeneous Multimedia Retrieval), which uses low cost to store and retrieve semantic information from heterogeneous multimedia data. Firstly, the particularity of heterogeneous multimedia retrieval in big data environments is addressed. Secondly, an approach to extract and represent semantic information for heterogeneous multimedia documents is proposed. Thirdly, a NoSQL-based approach to semantic storage, in which multimedia can be parallel processed in distributed nodes is provided. Finally, a MapReduce-based retrieval algorithm is presented and a user feedback supported scheme to achieve high retrieval precision and good user experience is designed. The experimental results indicate that the retrieval performance and economic efficiency of SHMR are suitable for multimedia information retrieval in heterogeneous big data environments.

Stackelberg Game for Bandwidth Allocation in Cloud-Based Wireless Live-Streaming Social Networks

Multimedia social networks have been introduced as a new technology to enrich people's lives through enhanced multimedia distribution. On the other hand, a media cloud system can perform multimedia processing and storage, and provide heterogeneous multimedia services. However, the challenges still remain for end users (e.g., mobile devices and PCs) to receive multimedia streaming from the cloud system with satisfied quality-of-service (QoS). To address these challenges, an efficient multimedia distribution approach taking advantage of live-streaming social networks is innovated in this paper to deliver the media services from the cloud to both desktop and wireless end users. Our approach allows bandwidth limited mobile users to acquire live multimedia streaming from desktop users, directly based on their social relationships rather than from the cloud. When a number of mobile users compete for limited bandwidth access with the desktop users, a bandwidth allocation problem must be solved to meet all users' QoS requirements in the live-streaming social network. We formulate the problem as a two-stage Stackelberg game, in which both desktop users and mobile users target at maximizing their utilities. In our study, a noncooperative game is used to model the competition among the desktop users in terms of shared bandwidth and price in the first stage of the game. The second stage of the game models the behavior of a mobile user selecting the desktop users by an evolutionary game. In addition, a case study is conducted following the general Stackelberg game formulation, where the existence of a unique Nash equilibrium is proved. Based on our game modeling, we design protocols for both desktop and mobile users and evaluate them with numerical examples.

Erratum to: A Load-aware Scheme for Providing Heterogeneous Multimedia Broadcast/Multicast Service (Het-MBMS)

Erratum to: A Load-aware Scheme for Providing Heterogeneous Multimedia Broadcast/Multicast Service (Het-MBMS)

Study of QoS for Heterogeneous Wireless Multimedia Sensor Networks

Study of QoS for Heterogeneous Wireless Multimedia Sensor Networks

A Load-aware Scheme for Providing Heterogeneous Multimedia Broadcast/Multicast Service (Het-MBMS)

As the utilization efficiency of network resource arouse the interests of researchers, Multimedia Broadcast Multicast Service (MBMS) with its inherent nature brings mobile wireless communications enter a new era. On the other hand, Next Generation Wireless Network (NGWN) trends to integrate multiple Radio Access Technologies (RATs) to optimize the overall network performance. In case, handover performance becomes a vital indicator of mobility experience of User Equipment (UE). Contrast to conventional vertical handover, the problem we facing is how to seamlessly handover for MBMS to provide heterogeneous MBMS (Het-MBMS). In order to address this problem, this paper proposes a new functional entity called Media Independent Broadcast Multicast Service Center (MIBM-SC), by the extensions and enhancements of Broadcast/Multicast-Service Center (BM-SC) and Media Independent Information Service (MIIS). Additionally, a network selection and a cell transmission mode selection schemes are proposed. Both schemes are based on a load-aware network capacity estimation algorithm to select the best target network and the best cell transmission mode. NS-2 based simulation validates that the proposed scheme enables to provide Het-MBMS and it improves the handover performance in terms of packet loss rate, throughput, handover delay, cell load, and bandwidth usage and Peak Signal-to-Noise Ratio (PSNR). Meanwhile, the total consumption of downlink power is reduced.

A Prioritized Adaptive Scheme for Multimedia Services over IEEE 802.11 WLANs

IEEE 802.11e protocol enables QoS differentiation between different traffic types, but requires MAC layer support and assigns traffic with static priority. This paper proposes an intelligent Prioritized Adaptive Scheme (iPAS) to provide QoS differentiation for heterogeneous multimedia delivery over wireless networks. iPAS assigns dynamic priorities to various streams and determines their bandwidth share by employing a probabilistic approach-which makes use of stereotypes. Unlike existing QoS differentiation solutions, the priority level of individual streams in iPAS is variable and considers service types and network delivery QoS parameters (i.e. delay, jitter, and packet loss rate). A bandwidth estimation technique is adopted to provide network conditions and the IEEE 802.21 framework is used to enable control information exchange between network components without modifying existing MAC protocol. Simulations and real life tests demonstrate how better results are obtained when employing iPAS than when either IEEE 802.11 DCF or 802.11e EDCA mechanisms are used. The iPAS key performance benefits are as follows: 1) better fairness in bandwidth allocation; 2) higher throughput than 802.11 DCF and 802.11e EDCA with up to 38% and 20%, respectively; 3) enables definite throughput and delay differentiation between streams.

Heterogeneous Metric Learning with Joint Graph Regularization for Cross-Media Retrieval

As the major component of big data, unstructured heterogeneous multimedia content such as text, image, audio, video and 3D increasing rapidly on the Internet. User demand a new type of cross-media retrieval where user can search results across various media by submitting query of any media. Since the query and the retrieved results can be of different media, how to learn a heterogeneous metric is the key challenge. Most existing metric learning algorithms only focus on a single media where all of the media objects share the same data representation. In this paper, we propose a joint graph regularized heterogeneous metric learning (JGRHML) algorithm, which integrates the structure of different media into a joint graph regularization. In JGRHML, different media are complementary to each other and optimizing them simultaneously can make the solution smoother for both media and further improve the accuracy of the final metric. Based on the heterogeneous metric, we further learn a high-level semantic metric through label propagation. JGRHML is effective to explore the semantic relationship hidden across different modalities. The experimental results on two datasets with up to five media types show the effectiveness of our proposed approach.

Modelling and Analysis of an Integrated Scheduling Scheme with Heterogeneous LRD and SRD Traffic

Multimedia applications in wireless networks are usually categorized into various classes according to their traffic patterns and differentiated Quality-of-Service (QoS) requirements. The traffic of heterogeneous multimedia applications often exhibits Long-Range Dependent (LRD) and Short-Range Dependent (SRD) properties, respectively. The integrated scheduling scheme that combines Priority Queuing (PQ) and Generalized Processor Sharing (GPS) within a hierarchical structure, referred to as PQ-GPS, has been identified as an efficient mechanism for QoS differentiation in wireless networks and attracted significant research efforts. However, due to the high complexity and interdependent relationship among traffic flows, modelling of the integrated scheduling scheme poses great challenges. To address this challenging and important research problem, we develop an original analytical model for PQ-GPS systems under heterogeneous LRD and SRD traffic. A cost-effective flow decomposition approach is proposed to equivalently divide the integrated scheduling system into a group of Single-Server Single-Queue (SSSQ) systems. The expressions for calculating the queue length distribution and loss probability of individual traffic flows are further derived. After validating its accuracy, the developed model is adopted as an efficient performance tool to investigate the important issues of resource allocation and call admission control in the integrated scheduling system under QoS constraints.

Development of Serious Games for Music Education

Serious games have proved to be an effective educational tool in many fields. The first goal of this paper is to illustrate some possible applications to music and their advantages. Moreover, music can be characterized by heterogeneous multimedia contents. Among the different facets music information is made of it is worth citing music symbols, their graphical representations as scores, their audio renderings as tracks, etc. The international standard known as IEEE 1599 is an XML multilayer format for heterogeneous music contents, and describes such different aspects in an integrated and synchronized context. Making relationships among music contents explicit provides a potentially rich educational environment. Consequently, this paper discusses the concept of multilayer learning object, introduces the IEEE 1599 standard, and finally shows some applications and case studies.

A P2P Platform for Collaborative Aggregated Multimedia Sharing

Peer-to-peer technologies have emerged as a powerful and scalable communication model for large scale content sharing. However, they are not yet provided with optimized heterogeneous aggregated content management functionality since they lack rich semantic specifications. To overcome these shortcomings, we elaborated a reference model of P2P architecture for a dynamic aggregation, sharing and retrieval of heterogeneous multimedia contents (simple or aggregated). This architecture was mainly developed under the CAM4Home European research project and is fully based on the CAM4Home semantic metadata model. This semantic model relies on RDF (Resource Description Framework) and is rich (but simple enough), extensible and dedicated for the description of any kind of multimedia content. In this paper, we detail and evaluate an original semantic-based community network architecture for heterogeneous multimedia content sharing and retrieval. Within the presented architecture, multimedia contents are managed according to their associated CAM4Home semantic metadata through a structured P2P topology. This topology relies on a semantically enhanced DHT (Distributed Hash Table) and is also provided with an additional indexing system for offering semantic storage and search facilities and overcoming the problem of exact match keywords in DHTs.

A Reconfigurable Heterogeneous Multimedia Processor for IC-Stacking on Si-Interposer

This paper presents a heterogeneous multimedia processor for embedded media applications such as image processing, vision, 3-D graphics and augmented reality (AR), assuming integrated circuit (IC)-stacking on Si-interposer. This processor embeds reconfigurable output drivers for external memory interface to increase memory bandwidth even in a mobile environment. The implemented output driver reconfigures its driving strength according to channel loss between the implemented processor and the memory, so it enables highspeed data communication while achieving 8× higher memory bandwidth compared to previous embedded media processors. The implemented processor includes three main programmable intellectual properties, mode-configurable vector processing units (MCVPUs), a unified filtering unit (UFU), and a unified shader. MCVPUs have 32 integer (16 bit) cores in order to support dual-mode operations between image-level processing and graphics processing. This mode-configuration enables a frame-level pipelining in AR application, so the proposed processor achieves 1.7× higher frame rate compared to the sequential AR processing. UFU supports 16 types of filtering operations only with a single instruction. Most image-level processing consists of various types of filtering operations, so UFU can improve media processing performance and energy-efficiency. UFU also supports texture filtering which is performance bottleneck of common graphics pipeline. A memory-access-efficient (off-chip memory) texturing algorithm named as an adaptive block selection is proposed to enhance texturing performance in 3-D graphics pipeline. UFU has two-level on-chip memory hierarchies, a 512B level-0 (L0) data buffer, and an 8kB level-1 (L1) static random-access memory (SRAM) cache. The small-sized L0 data buffer limits direct references to the large-sized L1 SRAM cache to reduce energy consumed in on-chip memories. Unified shader consists of four homogeneous scalar processing elements (SPEs) for geometry operations in 3-D graphics. Each SPE has single-precision floating-point data-paths, since precision of geometry operations in 3-D graphics is important in today's handheld devices (high resolution). The proposed media processor is fabricated in 0.13 μm CMOS technology with 4 mm × 4 mm chip size, and dissipates 275 mW for full AR operation.

Modeling and Analysis of a Hybrid CAC Scheme in Heterogeneous Multimedia Wireless Networks

A hybrid call admission control (CAC) scheme for a heterogeneous multimedia mobile network is proposed and investigated. In addition to the reservation of the system capacity to the cells, the network center also reserves an amount of bandwidth for a possible dynamic allocation to all cells in order to increase the system utilization and minimize the blocking probability of calls. In each cell, a specific class of calls may need a specific bandwidth for connection, a specific call connection time period, and may possess a specific mobility feature. By considering the specific characteristics for different classes of calls, each cell of the network has the capability to provide service to at least a minimum number of calls for each class. This paper successfully derives an analytical expression for the stationary distribution of the number of calls in the whole network and obtains explicit mathematical formulae for measuring system performance, including the utilization of the network, the blocking probability for each class of calls, and the throughput of the network. The consistency of the numerical analysis with the theoretical results and the simulation analysis based on the model description confirms the correctness of the modeling and analysis.