Heterogeneous Multimedia Research Articles

Using QoE Metric as a Decision Criterion in Multimedia Heterogeneous Network Optimization: Challenges and Research Perspectives

This article explores the growing importance of the QoE (Quality of Experience) metric as a fundamental criterion in the optimization of heterogeneous multimedia networks. We explore the benefits of using QoE, such as improving user experience, efficient resource management, and adaptation to network conditions. However, this paradigm presents technical challenges, including accurately measuring QoE and managing the complexity of heterogeneous networks. To address these challenges, we highlight promising research prospects, such as the development of advanced algorithms, real-time measurement of QoE, and the integration of machine learning. Ultimately, the use of QoE in the optimization of heterogeneous multimedia networks contributes to a substantial improvement in the quality of multimedia services offered to end users.

Multi-Agent Deep Reinforcement Learning for content caching within the Internet of Vehicles

In the Internet of Vehicles (IoV), multiple applications and multimedia services are deployed to enhance the quality of traveling experienced by passengers. These software components ingest heterogeneous multimedia content (e.g., streams, safety information, etc.) acquired through the Internet. However, the high mobility constraints and time-variant dynamics of the IoV may delay the acquisition process. Caching lowers acquisition delays of the content by storing requested content in the vicinity of its consumers.We propose a Multi-Agent Deep Reinforcement Learning (MADRL) based framework to design an optimal caching strategy for IoVs. The framework objective is to incentivize sharing caching resources while minimizing errors in estimating content needs. Extensive simulation results with different system parameters demonstrate the efficiency of the proposed solution.

V4Design: Intelligent Analysis and Integration of Multimedia Content for Creative Industries

Nowadays, vast amounts of multimedia content are being produced, archived, and digitized, resulting in great troves of data of interest. Examples include user-generated content, such as images, videos, text, and audio posted by users on social media and wikis, or content provided through official publishers and distributors, such as digital libraries, organizations, and online museums. This digital content can serve as a valuable source of inspiration to the creative industries, such as architecture and gaming, to produce new innovative assets or to enhance and (re-)use existing ones. However, in its current form, this content is difficult to be reused and repurposed due to the lack of appropriate solutions for its retrieval, analysis, and integration into the design process. In this article, we present V4Design, a novel framework for the automatic content analysis, linking, and seamless transformation of heterogeneous multimedia content to help architects and virtual reality game designers establish innovative value chains and end-user applications. By integrating and intelligently combining state-of-the-art technologies in computer vision, 3-D generation, text analysis, generation and semantic integration, and interlinking, V4Design provides architects and video game designers with innovative tools to draw inspiration from archive footage and documentaries, inspiring and eventually supporting the design process.

Forwarding and caching in video streaming over ICSDN: A clean-slate publish-subscribe approach

Nowadays, Internet usage has become prevalent, primarily because of high-quality heterogeneous multimedia content expectations from the subscriber (consumer), which puts tremendous pressure on the publisher (producer) in the networks. Information-Centric Networking (ICN) is a future internet architecture that optimizes data resources through content-based forwarding and caching, making it well-suited for multimedia content and video streaming (VS) scenarios. However, real-time data delivery is challenging in the current ICN-based publish–subscribe (pub-sub) mechanism, which pushes the existing pub-sub studies to prioritize more on the forwarding information base (FIB) rather than the pending interest table (PIT). This leads to issues such as inefficient caching and forwarding mechanisms, high overhead, and communication costs. To address these challenges, in this paper, we present a novel forwarding and caching solution named VS-ICSDN, integrating the combined principles of ICN-based pub-sub scheme and software-defined networking (SDN) in order to utilize the network resources more efficiently. We design a clean-slate caching strategy and name-based forwarding method to support both on-path and off-path caching on ICN nodes to coordinate flow entries among the SDN controller and clean-slate ICN nodes to maximize PIT utilization. In addition, the framework allows the content to be stored and searched in chunks with a single request to access the desired content, reducing the communication overhead and significantly improving overall performance. A simulation-based testbed and experimental result analysis validate our proposed work’s effectiveness in ensuring efficient network resource usage with low communication overhead and computational cost compared to other baseline methods.

Performance enhancement of multi channel multi beam FSO communication link with the application of Reed Solomon codes

The rising user needs for high speed applications and services have necessitated the requirement of a communication link that can provide the data rate comparable to optical fiber. Free Space Optical (FSO) Communication is an admired means of communication from past few years. To achieve seamless and error free transmission at high speed for heterogeneous multimedia applications at desired bit rates, novel advancements in optical communication technology are being explored along with FSO. However, several constraints such as atmospheric turbulence, uneven terrain, dynamic climatic conditions cause several nonlinearities which limit the connectivity. In this work, a multi channel multi beam FSO communication system by incorporating Reed Solomon (RS) Codes has been proposed at 30 Gbps. The Modified Gamma-Gamma model is being applied for channel turbulences. The application of Forward Error Correction (FEC) coding in the designed system offers ease of error detection and correction with improved performance at higher data rate for implementation over longer link lengths.The system performance has been evaluated for (i) Number of errors corrected, (ii) Bit Error Rate (BER), (iii) Received power, (iv) Geometric path losses and (v) Signal to Noise Ratio (SNR). At an acceptable BER of 10–9, the RS (255, 127, 129) encoded Four channel Four beam FSO communication link is effectively implemented for a distance of 3500 m with − 32.5 dBm received low optical power and 25 dB SNR.

Використання систем організації знань на основі онтологій у WIKI-ресурсах

The paper considers the theoretical foundations of knowledge organization systems (KOSs) in intelligent ontology-based applications. The aim of this study is to analyze the use of different types of KOSs to organize and improve the knowledge base of semantic Wiki resources that contain heterogeneous multimedia content of large volume and have a complex structure integrated knowledge from different domains. The dialects of the OWL ontology representation language and their expressiveness for representing special cases of ontologies used in KOSs are considered. The criteria for the classification of KOSs and sphere of their usage are analyzed. Formal model of ontology for semantic Wiki resource is proposed. This model is integrated with various implementing means for different types of relations between objects in the Semantic MediaWiki environment based on templates. Problems of access and retrieval of information in these resources and methods of their solving from the KOSs point of view are considered. The software implementation of the proposed approach with the example of the portal version of the Great Ukrainian Encyclopedia (e-VUE) is realized. The urgency of the problem intensifies by the need for national information resources in martial law situation, for which the determining factors of effective information processing are both the ability to obtain satisfaction of complex information needs and the relevance of the information obtained. This increases the importance of official government portals that integrate reliable data from various fields of knowledge and prevent possible misrepresentation (both accidental and malicious) of information in resources with open content generation.

Improving resource-constrained IoT device lifetimes by mitigating redundant transmissions across heterogeneous wireless multimedia of things

In Wireless Multimedia Sensor Networks (WMSNs), nodes capable of retrieving video, audio, images, and small scale sensor data, tend to generate immense traffic of various types. The energy-efficient transmission of such a vast amount of heterogeneous multimedia content while simultaneously ensuring the quality of service and optimal energy consumption is indispensable. Therefore, we propose a Power-Efficient Wireless Multimedia of Things (PE-WMoT), a robust and energy-efficient cluster-based mechanism to improve the overall lifetime of WMSNs. In a PE-WMoT, nodes declare themselves Cluster Heads (CHs) based on available resources. Once cluster formation and CH declaration processes are completed, the Sub-Cluster (SC) formation process triggers, in which application base nodes within close vicinity of each other organize themselves under the administration of a Sub-Cluster Head (SCH). The SCH gathers data from member nodes, removes redundancies, and forwards miniaturized data to its respective CH. PE-WMoT adopts a fuzzy-based technique named the analytical hierarchical process, which enables CHs to select an optimal SCH among available SCs. A PE-WMoT also devises a robust scheduling mechanism between CH, SCH, and child nodes to enable collision-free data transmission. Simulation results revealed that a PE-WMoT significantly reduces the number of redundant packet transmissions, improves energy consumption of the network, and effectively increases network throughput.

Efficient Share Generator for Slicing and Securely Retrieving the Cloud-Hosted Heterogeneous Multimedia Data

Recently, the security of heterogeneous multimedia data becomes a very critical issue, substantially with the proliferation of multimedia data and applications. Cloud computing is the hidden back-end for storing heterogeneous multimedia data. Notwithstanding that using cloud storage is indispensable, but the remote storage servers are untrusted. Therefore, one of the most critical challenges is securing multimedia data storage and retrieval from the untrusted cloud servers. This paper applies a Shamir Secrete-Sharing scheme and integrates with cloud computing to guarantee efficiency and security for sensitive multimedia data storage and retrieval. The proposed scheme can fully support the comprehensive and multilevel security control requirements for the cloud-hosted multimedia data and applications. In addition, our scheme is also based on a source transformation that provides powerful mutual interdependence in its encrypted representation—the Share Generator slices and encrypts the multimedia data before sending it to the cloud storage. The extensive experimental evaluation on various configurations confirmed the effectiveness and efficiency of our scheme, which showed excellent performance and compatibility with several implementation strategies.

A Residual Chaotic System for Image Security and Digital Video Watermarking

Multimedia data security has become a noticeable problem in the continuous development of multimedia technology. Traditional chaotic systems usually require larger bit-width iterative calculations to affect the generation rate of chaotic systems when generating. To solve this problem, we proposed a general Residual Chaotic System (RCS) for image security in heterogeneous multimedia signal processing environments. Based on the Residue Number System (RNS) commonly used in VLSI, we generate the general chaotic sequences by iterating unary polynomial function through it in Galois Field (GF). Moreover, we make use of the Box-Muller algorithm to generate RCS, which improved the speed of operation by a large margin when doing R/B backward transformation. Experiments show that the key sensitivity and plaintext sensitivity of the proposed encryption system are good, and the system can encrypt pictures efficiently and safely. Then, we proposed a watermark embedding and extraction algorithm with block selection. Based on the proposed new judgment conditions, the $4\times 4$ brightness blocks are divided into five categories, then the appropriate types of blocks are selected accordingly. After testing the performance indexes of cypher-text images, including statistical and sensitivity-related tests, all results indicate that our algorithm has good randomness and security. The invisibility is good, the watermark capacity has been significantly improved, and the video bit-rate growth rate is in a good range. The watermark can resist attacks such as noise and the bit error rate under re-encoding and Gaussian low-pass filtering attacks. Compared with other algorithms, the algorithm that we proposed is faster, much more efficient, and more secure. Therefore, it is a practical alternative scheme for heterogeneous communication environments.

SDN-Assisted DDoS Defense Framework for the Internet of Multimedia Things

The Internet of Things is visualized as a fundamental networking model that bridges the gap between the cyber and real-world entity. Uniting the real-world object with virtualization technology is opening further opportunities for innovation in nearly every individual’s life. Moreover, the usage of smart heterogeneous multimedia devices is growing extensively. These multimedia devices that communicate among each other through the Internet form a unique paradigm called the Internet of Multimedia Things (IoMT). As the volume of the collected data in multimedia application increases, the security, reliability of communications, and overall quality of service need to be maintained. Primarily, distributed denial of service attacks unveil the pervasiveness of vulnerabilities in IoMT systems. However, the Software Defined Network (SDN) is a new network architecture that has the central visibility of the entire network, which helps to detect any attack effectively. In this regard, the combination of SDN and IoMT, termed SD-IoMT , has the immense ability to improve the network management and security capabilities of the IoT system. This article proposes an SDN-assisted two-phase detection framework, namely SD-IoMT-Protector, in which the first phase utilizes the entropy technique as the detection metric to verify and alert about the malicious traffic. The second phase has trained with an optimized machine learning technique for classifying different attacks. The outcomes of the experimental results signify the usefulness and effectiveness of the proposed framework for addressing distributed denial of service issues of the SD-IoMT system.

Ontological methods and tools for semantic extension of the media WIKI technology

Practical aspects of ontological approach to organization of intelligent Wiki-based information resources (IR) are considered. We analyze the main features, capabilities and limitations of MediaWiki as a technological platform for development of the Web-based information resource and suggest main directions of its refinement. We propose an abstract model of MediaWiki architecture that formalizes relations between the main components of this software environment and analyze the ways of its semantic extensions based on ontological representation of domain knowledge. An original algorithm of semantic Wiki pages matching with domain ontology is developed. We propose an ontological model of IR that formalizes its knowledge base structure and explicitly performs main features of typical information objects (TIO) of this IR. Such TIOs depend on domain specifics and purposes of IR, therefore their development has to involve domain experts and knowledge engineers. Use of ontology corresponding to the set of Wiki pages (either with semantic markup or without it) provides new IR functions associated with semantic search and navigation. Other important aspect of intelligent Wiki resource development deals with adaptation of user interface to the specifics of IR: enabling various tools of navigation, visualization and content analysis by processing of TIO features enriches IR functionality, reduces access time to information and makes usage of IR more efficient. Developing additional MediaWiki functionality with new requests to the MediaWiki API using TIO templates, extends data analysis and integration capabilities, and offers different, user-focused, IR content views expands the possibilities of data integration and proposes various user-oriented representations of IR content. Wiki resource semantization allows the use knowledge acquired from such IR by external application, or example, by search engines for intelligent Web retrieval. Domain ontologies based on various subsets of the Wiki pages and generated by them thesauri can be used by various Semantic Web applications, both independently or in general technological chain for personified retrieval focused on individual users and their tasks. Approbation of this approach is demonstrated by MAIPS retrieval system. We consider the use semantic similarity of concepts represented by Wiki-pages of IR as an additional way of intelligent navigation between these pages. Such approach allows to group Wiki pages according to user interests by different aspects of their content and structure. Wiki ontologies are considered as the basis for estimation of semantic similarity between domain concepts pertinent to user task. Such elements of Wiki ontology as classes, property values of class instances and relations between them are used as parameters for the quantitative assessment of semantic similarity of Wiki pages. We propose to use local similarity and generate the sets of semantically similar concepts (SSC) that takes into account some subset of page properties and categories defined by user needs. Such sets of SSCs can be considered as user task thesauri for other applications. In addition, we propose to enrich the basic tools of MediaWiki used for access management to the IR content with specialized software code that performs content classification that take into consideration separate namespaces, categories, templates and semantic properties of TIO acquired from Wiki markup. We demonstrate the software implementation of proposed solutions by developing of portal version of the Great Ukrainian Encyclopedia (e-VUE) that contains heterogeneous multimedia content with complex structure. We analyze the specifics of e-VUE knowledge system and develop its formalized TIO representation based on Semantic Web technologies and ontological analysis. Ontological model of e-VUE and original methods of its processing used for this project extend the functionality of the portal in the area of search, navigation, integration and protection of content based on background domain knowledge. In addition, original user interface of e-VUE is developed with an allowance for Encyclopedia knowledge specifics, substantially differs from the standard Wiki, meets the requirements, goals and objectives of this IR and provides a lot of additional features.

Efficient meta-heuristic approaches in solving minimal exposure path problem for heterogeneous wireless multimedia sensor networks in internet of things

One of the well-known methods for evaluating Heterogeneous wireless multimedia sensor networks (HWMSNs) in Internet of Things have drawn attention of the research community because this type of networks possesses great advantages of both coverage and performance. One of the most fundamental issues in HWMSNs is the barrier coverage problem which evaluates the surveillance capability of the network systems, especially those designed for security purposes. Among multiple approaches to solve this issue, finding the minimal exposure path (MEP), which corresponds to the worst-case coverage of the network is the most popular and efficient way. However, the MEP problem in HWMSNs (hereinafter heterogeneous multimedia MEP or HM-MEP) is specifically complex and challenging with the unique features of the HWMSNs. Thus, the problem is then converted into numerical functional extreme with high dimension, non-differential and non-linearity. Adapting to these features, two efficient meta-heuristic algorithms, Hybrid Evolutionary Algorithm (HEA) and Gravitation Particle Swarm Optimization (GPSO) are proposed for solving the problem. The HEA is a hybrid evolutionary algorithm in combination with local search while the GPSO is a novel particle swarm optimization based on the gravity force theory. Experimental results on extensive instances indicate that the proposed algorithms are suitable for the HM-MEP problem and perform well in term of both solution accuracy and computation time compared to existing approaches.

An approach for assembly process case discovery using multimedia information source

Assembly process design (APD) is a time-consuming and error-prone activity of complex product manufacturing. Assembly process reuse can efficiently improve the process design quality and shorten design cycle. With the rapid development of digital technologies, the information source, which stores reusable information of an existing APD result, becomes multiple and heterogeneous multimedia, e.g. text, 3D model, even images and videos. Currently, most researches focused on using single and structured information source, there has been very little research on how to use multimedia information source. For reusing assembly process, one appropriate way is to capture assembly process case (APC) from existing APD result, retrieve an APC similar to the new assembly and modify the process of retrieved APC to generate assembly process of new assembly. In this paper, a novel approach is proposed to discover assembly process case for assembly process reuse using multimedia information source. First, the integrated representation of APC is constructed, which consists of hierarchical topological structure and sematic assembly process concept. Then, by multimedia dismantling and single-medium parsing, the assembly process information is extracted uniformly from multimedia information source. Moreover, the process similarity between new assembly and APC is calculated to retrieve the reusable APC. The process similarity calculation is comprehensively accomplished from two aspects: applying graph algorithm to calculate the topological structure similarity and applying weighted bipartite graph to calculate the assembly process concepts similarity. Finally, an illustrative case study is performed to analyze the effectiveness of the proposed approach. The result showed that the proposed approach can build an APC library from multimedia information source and help assembly process designers to discover APC for assembly process reuse.

Affective Computing for Large-scale Heterogeneous Multimedia Data

The wide popularity of digital photography and social networks has generated a rapidly growing volume of multimedia data (i.e., images, music, and videos), resulting in a great demand for managing, retrieving, and understanding these data. Affective computing (AC) of these data can help to understand human behaviors and enable wide applications. In this article, we survey the state-of-the-art AC technologies comprehensively for large-scale heterogeneous multimedia data. We begin this survey by introducing the typical emotion representation models from psychology that are widely employed in AC. We briefly describe the available datasets for evaluating AC algorithms. We then summarize and compare the representative methods on AC of different multimedia types, i.e., images, music, videos, and multimodal data, with the focus on both handcrafted features-based methods and deep learning methods. Finally, we discuss some challenges and future directions for multimedia affective computing.

Introduction to the Special Issue on Affective Computing for Large-scale Heterogeneous Multimedia Data

Introduction to the Special Issue on Affective Computing for Large-scale Heterogeneous Multimedia Data

A multimedia case-based reasoning framework for assembly sequence planning

PurposeAssembly sequence planning (ASP) is a crucial job during assembly process design. However, it is still difficult to reuse the existing solution to solve a new ASP problem. In particular, with the rapid development of digital technologies, the reusable assembly information of an existing solution is not concentrated in one multimedia but dispersed in multiple heterogeneous multimedia, e.g. text, three-dimensional graphics, even images and videos. This paper aims to propose a multimedia case (MC)-based reasoning framework to solve ASP by reusing the existing solution whose assembly information is dispersed in multimedia.Design/methodology/approachThe proposed framework is designed with the introduction of the MC. An MC seamlessly integrates the dispersed assembly information of an existing solution. Under the proposed concept and architecture, the assembly information of an existing solution is extracted to build assembly descriptors of multimedia. Therefore, the MC is captured by organizing the assembly descriptors of corresponding multimedia.FindingsBy means of the framework proposed, it is possible to reuse the existing solution whose assembly information is dispersed in multimedia to solve ASP. Moreover, the extraction method of assembly information can flexibly parse most of the multimedia. Finally, the MC has the capability to represent the existing solution by collecting dispersed assembly information.Originality/valueThe proposed framework can discover the similar existing solution and avoid the potential failures confronted in the past so that the feasibility of ASP result can be improved as much as possible.

Impact of Variable Packet Length on the Performance of Heterogeneous Multimedia Wireless Sensor Networks

Performance valuation of wireless sensor network (WSN) routing protocols calls for realistic data traffic models because of their application specific nature. Since energy limitations is a cardinal issue for WSNs, network lifetime has become a key performance metric. In the quest towards energy efficiency, data packet length is an essential factor, especially in the multimedia applications. On one hand, a modest packet length brings down the packet loss, nonetheless, it results in dissipation of more energy. Thus, there comes forth a tradeoff in resolving the data packet length, where both low and high bit rates lead to certain energy inefficiency problems. This paper analyses the effect of variable packet length, being majorly ignored by routing protocols, on the network lifetime, throughput and reliability. The presented work is validated using computer simulations.

Deep Semantic Mapping for Heterogeneous Multimedia Transfer Learning Using Co-Occurrence Data

Transfer learning, which focuses on finding a favorable representation for instances of different domains based on auxiliary data, can mitigate the divergence between domains through knowledge transfer. Recently, increasing efforts on transfer learning have employeddeepneuralnetworks (DNN) to learn more robust and higher level feature representations to better tackle cross-media disparities. However, only a few articles consider the correction and semantic matching between multi-layer heterogeneous domain networks. In this article, we propose adeep semantic mapping model forheterogeneous multimediatransferlearning (DHTL) using co-occurrence data. More specifically, we integrate the DNN withcanonicalcorrelationanalysis (CCA) to derive a deep correlation subspace as the joint semantic representation for associating data across different domains. In the proposed DHTL, a multi-layer correlation matching network across domains is constructed, in which the CCA is combined to bridge each pair of domain-specific hidden layers. To train the network, a joint objective function is defined and the optimization processes are presented. When the deep semantic representation is achieved, the shared features of the source domain are transferred for task learning in the target domain. Extensive experiments for three multimedia recognition applications demonstrate that the proposed DHTL can effectively find deep semantic representations for heterogeneous domains, and it is superior to the several existing state-of-the-art methods for deep transfer learning.

MHCP: Multimedia Hybrid Cloud Computing Protocol and Architecture for Mobile Devices

Multimedia cloud computing has appeared as a very attractive environment for the business world in terms of providing cost-effective services with a minimum of entry costs and infrastructure requirements. There are some architecture proposals in the related literature, but there is no multimedia cloud computing architecture with hybrid features specifically designed for mobile devices. In this article, we propose a new multimedia hybrid cloud computing architecture and protocol. It merges existing private and public clouds and combines IaaS, SaaS and SECaaS cloud computing models in order to find a common platform to deliver real time traffic from heterogeneous multimedia and social networks for mobile users. The developed protocol provides suitable levels of QoS, while providing a secure and trusted cloud environment.

A Big Data Layered Architecture and Functional Units for the Multimedia Internet of Things

The escalating growth of multimedia content in Internet of Things (IoT) applications leads to a huge volume of unstructured data being generated. Unstructured Big data has no particular format or structure and can be in any form such as text, audio, images, and video. Furthermore, current IoT systems cannot successfully realize the notion of having ubiquitous connectivity of everything if they are not capable to include ‘multimedia things’. In this paper, we address two issues by proposing a new architecture for the Multimedia Internet of Things (MIoT) with Big multimodal computation layer. We first introduce MIoT as a novel paradigm in which smart heterogeneous multimedia things can interact and cooperate with one another and with other things connected to the Internet to facilitate multimedia-based services and applications that are globally available to the users. The MIoT architecture consists of six layers. The computation layer is specially designed for Big multimodal analytics. This layer has four important functional units: Data Centralized Unit, Multimodal Data Aggregation Unit, Multimodal Data Divide & Conquer Computation Unit, and Fusion & Decision Making Unit. A novel and highly scalable technique called the Divide & Conquer Principal Component Analysis (DC-PCA) for feature extraction in the divide and conquer mechanism is proposed to be used together with the Divide & Conquer Linear Discriminant Analysis (DC-LDA) for multimodal Big data analytics. Experiments are conducted to confirm the good performance of these techniques in the functional units of the Divide & Conquer computational mechanisms. The final section of the paper gives application on a camera sensing IoT platform and real-world data analytics on multicore architecture implementations.