Use Of Multimedia Data Research Articles

Spatial-Aware and Load Balancing Distributed Data Partitioning Strategies for Content-Based Multimedia Retrieval.

Content-Based Multimedia Retrieval (CBMR) has become very popular in several applications, driven by the growing routine use of multimedia data. Since the datasets used in real-world applications are very large and descriptor's dimensionality is high, querying is an expensive, albeit important functionality. Further, exact search is prohibitive in most cases, motivating the use of Approximate Nearest Neighbour Search (ANNS) algorithms, trading accuracy for performance. These have been mainly developed targeting a sequential execution in a single node. However, the large and increasing datasets used and the high query loads submitted to those systems typically surpass the memory and computing resources available in a single node. This motivated the development of parallel distributed memory ANNS solutions to meet the computing capabilities required by those applications. A common problem that must be handled when using distributed memory systems is data partitioning and its impact on load imbalance. Several data partitioning approaches have already been proposed, including elaborated spatial-aware strategies. However, little effort has been put into carefully analyzing the performance of those strategies at scale. Here, we evaluated the commonly used data partitioning strategies in ANNS and identified their limitations to propose a novel class of partitioning algorithms that can minimize load imbalance while improving data locality to attain high performance on the distributed memory search. Experimentally, we found that our proposed algorithms (SABBS and SABBSR) improved search performance by up to 1.64× compared to the best previous solution. In a distributed memory weak scaling evaluation, with up to 12 billion 128-dimensional descriptors and 60 compute nodes, the gains were maintained as the system scaled with our novel approaches. These results demonstrate the efficiency of our new algorithms for billion-scale ANNS and the importance of considering not only data locality but also data and load imbalance in the data partitioning.

Effective Use of Multimedia Data Mining Assisted Teaching in College Football Training Instruction

Abstract Multimedia-assisted data mining algorithms are used in this paper to extract information from soccer training teaching data and develop models for the sampling and storing soccer training data. Filtering is used to process the soccer training attribute data stream, and global position optimization is performed using the mutual information analysis method to integrate the teaching system resources. The soccer training teaching model and teaching program were established to finish the teaching system design. The results show that after one semester of teaching, the final assessment, compared to the average score of the entrance assessment, the training performance increased by about 42%, thus proving the effectiveness of the soccer training teaching system based on multimedia-assisted teaching.

The Face Deepfake Detection Challenge.

Multimedia data manipulation and forgery has never been easier than today, thanks to the power of Artificial Intelligence (AI). AI-generated fake content, commonly called Deepfakes, have been raising new issues and concerns, but also new challenges for the research community. The Deepfake detection task has become widely addressed, but unfortunately, approaches in the literature suffer from generalization issues. In this paper, the Face Deepfake Detection and Reconstruction Challenge is described. Two different tasks were proposed to the participants: (i) creating a Deepfake detector capable of working in an “in the wild” scenario; (ii) creating a method capable of reconstructing original images from Deepfakes. Real images from CelebA and FFHQ and Deepfake images created by StarGAN, StarGAN-v2, StyleGAN, StyleGAN2, AttGAN and GDWCT were collected for the competition. The winning teams were chosen with respect to the highest classification accuracy value (Task I) and “minimum average distance to Manhattan” (Task II). Deep Learning algorithms, particularly those based on the EfficientNet architecture, achieved the best results in Task I. No winners were proclaimed for Task II. A detailed discussion of teams’ proposed methods with corresponding ranking is presented in this paper.

A Highly Secured Image Encryption Scheme using Quantum Walk and Chaos

The use of multimedia data sharing has drastically increased in the past few decades due to the revolutionary improvements in communication technologies such as the 4th generation (4G) and 5th generation (5G) etc. Researchers have proposed many image encryption algorithms based on the classical random walk and chaos theory for sharing an image in a secure way. Instead of the classical random walk, this paper proposes the quantum walk to achieve high image security. Classical random walk exhibits randomness due to the stochastic transitions between states, on the other hand, the quantum walk is more random and achieve randomness due to the superposition, and the interference of the wave functions. The proposed image encryption scheme is evaluated using extensive security metrics such as correlation coefficient, entropy, histogram, time complexity, number of pixels change rate and unified average intensity etc. All experimental results validate the proposed scheme, and it is concluded that the proposed scheme is highly secured, lightweight and computationally efficient. In the proposed scheme, the values of the correlation coefficient, entropy, mean square error (MSE), number of pixels change rate (NPCR), unified average change intensity (UACI) and contrast are 0.0069, 7.9970, 40.39, 99.60%, 33.47 and 10.4542 respectively.

Consumer Behavior in the Online Classroom: Using Video Analytics and Machine Learning to Understand the Consumption of Video Courseware

Video is one of the fastest growing online services offered to consumers. The rapid growth of online video consumption brings new opportunities for marketing executives and researchers to analyze consumer behavior. However, video also introduces new challenges. Specifically, analyzing unstructured video data presents formidable methodological challenges that limit the use of multimedia data to generate marketing insights. To address this challenge, the authors propose a novel video feature framework based on machine learning and computer vision techniques, which helps marketers predict and understand the consumption of online video from a content-based perspective. The authors apply this framework to two unique data sets: one provided by MasterClass, consisting of 771 online videos and more than 2.6 million viewing records from 225,580 consumers, and another from Crash Course, consisting of 1,127 videos focusing on more traditional education disciplines. The analyses show that the framework proposed in this article can be used to accurately predict both individual-level consumer behavior and aggregate video popularity in these two very different contexts. The authors discuss how their findings and methods can be used to advance management and marketing research with unstructured video data in other contexts such as video marketing and entertainment analytics.

Design implementation of wireless multimedia sensor networks for dryland agriculture

This study discusses the application of Wireless Sensor Network (WSN) technology to specific dry land agriculture because it was founded in the Province of East Nusa Tenggara (NTT) and a small part of Maluku. Its uniqueness has become a comparative advantage for the regions, especially for the academics of the University of Nusa Cendana (Undana). In general, the use of WSN in agriculture is still basic, relying on analog and manual sensors to obtain information on land conditions and vegetation in the land. Also generally still using scalar data and not optimizing the use of multimedia data, especially data based on videos and images. The contribution and novelty of this research are: (a) The application of multi-sensor technology with the Wireless Multimedia Sensor Network (WMSN) will optimize various data from land conditions so that it is hoped that more analysis of monitoring data results is expected, precise and efficient; (b) The application of WMSN with Internet-of-Thing (IoT) will be applied to the dry land agriculture. This research produces a design for implementing WMSN technology on dry agricultural land which is expected to increase agricultural yields in the area.

Consumer Behavior in the Online Classroom: Using Video Analytics and Machine Learning to Understand the Consumption of Video Courseware

Video is one of the fastest growing online services offered to consumers. The rapid growth of online video consumption brings new opportunities for marketing executives and researchers to analyze consumer behavior. However, video also introduces new challenges. Specifically, analyzing unstructured video data presents formidable methodological challenges that limit the current use of multimedia data to generate marketing insights. To address this challenge, the authors propose a novel video feature framework based on machine learning and computer vision techniques, which helps marketers predict and understand the consumption of online video from a content-based perspective. The authors apply this frame- work to two unique datasets: one provided by MasterClass, consisting of 771 online videos and more than 2.6 million viewing records from 225,580 consumers, and another from Crash Course, consisting of 1,127 videos focusing on more traditional education disciplines. The analyses show that the framework proposed in this article can be used to accurately predict both individual-level consumer behavior and aggregate video popularity in these two very different contexts. The authors discuss how their findings and methods can be used to advance management and marketing research with unstructured video data in other contexts such as video marketing and entertainment analytics.

A lattice-based searchable encryption scheme with the validity period control of files

In recent years, with the dramatic increase in the use of multimedia data, rapid retrieval and sharing of the multimedia data have become major trends. The validity period control function widely used in daily softwares, which enables multimedia data to be shared from a specific time and revokes the authorization of shared data at another specific time, thereby effectively enhancing the personalized experience of multimedia data users. At the same time, to protect privacy, databases usually store data in an encrypted form and use searchable encryption technology to retrieve keywords on the ciphertext to quickly extract the required multimedia data. However, the existing searchable encryption scheme cannot control the reading of multimedia data by using validity period. To solve this problem, we propose a lattice-based searchable encryption scheme with the validity period control of files. In order that the data owners have flexible and fine-grained control over the files, we convert the time information into a lattice vector for setting a different valid time for each file. Meanwhile, we embed the time information into the ciphertext, so that the data owner only needs to set the validity period once to automatically maintain the life cycle of the files over encrypted data, instead of manually withdrawing each expired file. Furthermore, we combine searchable encryption and time-distance so that our scheme returns valid files only if the keywords match correctly and the search time matches the file’s validity period. This scheme is a candidate for time-controlled searchable encryption technology in the post-quantum era because our scheme constructed on learning with error (LWE) problem which has been proven to resist quantum attacks.

A Hybrid Modified Advanced Encryption Standard and Chaos Encryption Algorithm for Securing Compressed Multimedia Data

Multimedia data is being used more widely in the field of Internet communication technology. Over wireless networks, extensive use of multimedia applications can easily be intercepted. The widespread use of multimedia data makes media content security and protection increasingly urgent and necessary. In order to fully protect or maintain security and privacy of multimedia data, it must be secured through encryption before transmission. Multimedia encryption is the core enabling technology that provides confidentiality and end-to-end security preventing unauthorized access of the information. In this paper, an encryption algorithm is proposed to achieve the desired integrity. Compressed video frames usingH.264 with multi-view video coding MVC are encrypted using a hybrid algorithm based on modified advanced encryption standard AES with 1D, and 2D chaotic maps. The encryption process replaces add round key and S-box operations of AES encryption by XOR operation and chaotic map generators on different blocks of input data frames. The results of our simulation using MATLAB show that the proposed algorithm is suitable for multimedia encryption applications.

Encryption techniques for H.264/AVC videos: A literature review

There are tremendous impacts of advancement in information and telecommunication technology over the use of multimedia data. Multimedia data especially videos are stored and transmitted through the wired and wireless network, where they can be easily accessed. Thus, safeguarding the video data from unauthorized access and protecting the integrity and authenticity has become mandatory for transmission as well as storage. Encryption is one of the best technologies in Digital Rights Management (DRM) scheme which is used to protect copyright works. This paper reviews some previous and recent works for encryption techniques of H.264/AVC video encoders. The reviewed work has been categorized based on the area where the encryption scheme is applied.

Learning Dynamic Stream Weights For Coupled-HMM-based Audio-visual Speech Recognition

With the increasing use of multimedia data in communication technologies, the idea of employing visual information in automatic speech recognition (ASR) has recently gathered momentum. In conjunction with the acoustical information, the visual data enhances the recognition performance and improves the robustness of ASR systems in noisy and reverberant environments. In audio-visual systems, dynamic weighting of audio and video streams according to their instantaneous confidence is essential for reliably and systematically achieving high performance. In this paper, we present a complete framework that allows blind estimation of dynamic stream weights for audio-visual speech recognition based on coupled hidden Markov models (CHMMs). As a stream weight estimator, we consider using multilayer perceptrons and logistic functions to map multidimensional reliability measure features to audiovisual stream weights. Training the parameters of the stream weight estimator requires numerous input-output tuples of reliability measure features and their corresponding stream weights. We estimate these stream weights based on oracle knowledge using an expectation maximization algorithm. We define 31-dimensional feature vectors that combine model-based and signal-based reliability measures as inputs to the stream weight estimator. During decoding, the trained stream weight estimator is used to blindly estimate stream weights. The entire framework is evaluated using the Grid audio-visual corpus and compared to state-of-the-art stream weight estimation strategies. The proposed framework significantly enhances the performance of the audio-visual ASR system in all examined test conditions.

Robust Keypoint Detection Using Higher-Order Scale Space Derivatives: Application to Image Retrieval

Image retrieval has been extensively studied over the last two decades due to the increasing demands for the effective use of multimedia data. Among various approaches to image retrieval, scale space representation and local keypoint descriptors have been shown to be a promising approach. Even though the concept of scale space representation has been known for a long time, it has now gained prominence as a powerful method for image retrieval mostly due to the invention of the Scale Invariant Feature Transform (SIFT). We will review the characteristics of the scale space operation and provide an extended method of scale space operation that significantly improves the image matching accuracy in the context of image retrieval. We use an operational tattoo image database containing 1,000 near duplicate images to show the superior retrieval performance of the proposed method compared to SIFT keypoints.

Rules Generation for Multimedia Data Classifying using Rough Sets Theory

An efficient multimedia data management process is very important in multimedia system application. Huge size of multimedia data that distributed in multi locations makes multimedia data management more complicated. Rapid development of multimedia applications created a vast volume of multimedia data and it is exponentially incremented from time to time. This situation requires for efficient data classification and organization technique for providing effective multimedia data manipulation process. Using rough sets theory and web services technology, this paper proposed a new rules generation for multimedia data classifying in collaborative environment. ROSETTA tool is applied to verify the reliability of the generated results. The experiments show that the rough sets theory based for multimedia data classifying is suitable to be executed in web services environment.

New Advance Encryption Standard to Analyze Encrypted Image Quality

Modern advancement in information technology has enabled pervasive use of multimedia data in variety of applications. Due to increased use of multimedia data on internet it is desired to secure the confidential multimedia data. This paper proposes a new encryption scheme as New Advance Encryption Standard for image encryption. AES is based on substitutionpermutation network which is fast for both software and hardware. The New Advance Encryption Standard (NAES) proposes a new encryption scheme by introducing more potential shift row transformation than that of original AES. This NAES allows for more security and better image quality measures. Experimental results are also provided to demonstrate the effectiveness of proposed encryption scheme. General Terms Cryptography, Encryption, Decryption, Image Encryption.

Trial Realization of Human-Centered Multimedia Navigation for Video Retrieval

A trial realization of human-centered navigation for video retrieval is presented in this article. This system consists of the following functions: (a) multimodal analysis for collaborative use of multimedia data, (b) preference extraction for the system to adapt to users' individual demands, and (c) adaptive visualization for users to be guided to their desired contents. By using these functions, users can find their desired video contents more quickly and accurately than with the conventional retrieval schemes since our system can provide new pathways to the desired contents. Experimental results verify the effectiveness of the proposed system.

The Effect of Speed Variation on Different Traffic Patterns in Mobile Ad Hoc Network

The constant bit rate (CBR) traffic which generates data packets at a constant rate is very well known traffic model for mobile ad-hoc network(MANET). This type of traffic is good enough for text data packets but not so for multimedia applications as they generate traffic in a burst followed by some idle periods. Increasing use of multimedia data prompts us to investigate some other traffic models which can accommodate the specific features of these data. In this articles, we consider two such traffic patterns (i) Exponential traffic model which is based on exponential distribution and (ii) Pareto traffic model based on Pareto distribution. We have tried to study the behavior of different mobile ad-hoc network routing protocols under all traffic sources for nodes moving with different mobility models with varying speed of the mobile nodes. We found that all the characteristics perform better for CBR traffic than Exponential and Pareto traffic for Normalized Routing Load (NRL), Packet Delivery Fraction (PDF) and Throughput as the existing routing protocols are designed for such traffic.

Classification of audio signals using generalized spatial fuzzy clustering.

With the increasing use of multimedia data, the need for automatic classification and retrieval of certain kinds of audio data has become an important issue. In this paper, we propose an efficient method of audio signal segmentation and classification from audiovisual database. While conventional methods apply thresholding to audio features such as energy and zero‐crossing rate to detect the boundaries, causing misclassification for audio signals which contain certain audio effects such as fade‐in, fade‐out, and cross‐fade, the proposed algorithm, called general spatial fuzzy c‐means algorithm (GSFCM), solves the problem by taking into account the local spatial information which is weighted correspondingly to neighbor elements based on their distance attributes. GSFCM detects the boundaries between two different audio signals, classifies segments, and then extracts unique feature vectors. This results in the accurate detection and classification. Experiment results for the audio signal from TV news program at 44.1 kHz with 30‐min long confirm that the proposed method outperforms conventional methods in terms of accuracy of the audio signal classification. These results demonstrate that the proposed method is a suitable candidate for audio‐video indexing which is compressed by MPEG. [Work supported by the MKE, Korea, under the ITRC supervised by IITA (IITA‐2008‐(C1090‐0801‐0039)).]

Mapping, indexing and querying of MPEG-7 descriptors in RDBMS with IXMDB

MPEG-7 is a promising standard for the description of multimedia content. A number of applications based on MPEG-7 media descriptions have been set up for research, commercial and industrial applications. Therefore, an efficient storage solution for large amounts of MPEG-7 descriptions is certainly desirable. As a kind of data-centric XML documents, MPEG-7 descriptions can be stored in the relational DBMS for efficient and effective management. The approaches of storing XML data in relational DBMS can be classified into two classes of storage model: schema-conscious and schema-oblivious. The schema-conscious model, however, cannot support complex XPath-based queries efficiently and the schema-oblivious approach lacks the flexibility in typed representation and access. Although the leading database systems have provided functionality for the XML document management, none of them can reach all the critical requirements for the MPEG-7 descriptions management. In this paper, we present a new storage approach, called IXMDB, for MPEG-7 documents storage solution. IXMDB integrates the advantages of both the schema-conscious method and the schema-oblivious method, and avoids the main drawbacks from each method. The design of IXMDB pays attention to both multimedia information exchange and multimedia data manipulation. Its features can reach the most critical requirements for the MPEG-7 documents storage and management. The translation mechanism for converting XQuery to SQL and the support of query from multimedia perspective are provided with IXMDB. Performance studies are conducted by performing a set of queries from the XML perspective and from the multimedia perspective. The experimental results are presented in the paper and initial results are encouraging.

Similarity‐Based Clustering Strategy for Mobile Ad Hoc Multimedia Databases

Multimedia data are becoming popular in wireless ad hoc environments. However, the traditional content‐based retrieval techniques are inefficient in ad hoc networks due to the multiple limitations such as node mobility, computation capability, memory space, network bandwidth, and data heterogeneity. To provide an efficient platform for multimedia retrieval, we propose to cluster ad hoc multimedia databases based on their semantic contents, and construct a virtual hierarchical indexing infrastructure overlaid on the mobile databases. This content‐aware clustering scheme uses a semantic‐aware framework as the theoretical foundation for data organization. Several novel techniques are presented to facilitate the representation and manipulation of multimedia data in ad hoc networks: 1) using concise distribution expressions to represent the semantic similarity of multimedia data, 2) constructing clusters based on the semantic relationships between multimedia entities, 3) reducing the cost of content‐based multimedia retrieval through the restriction of semantic distances, and 4) employing a self‐adaptive mechanism that dynamically adjusts to the content and topology changes of the ad hoc networks. The proposed scheme is scalable, fault‐tolerant, and efficient in performing content‐based multimedia retrieval as demonstrated in our combination of theoretical analysis and extensive experimental studies.

On the Need for Visual Formalisms in Music Processing

Computer music environments (CMEs) are notoriously difficult to design and implement. As computer programs, they reflect the complex nature of music ontology and must support real-time manipulation of multimedia data. In addition, these programs must be usable by native users, supporting their creative process without obstructing it through technical difficulties. To achieve these goals, the authors argue, CMEs must be provided with a well-defined methodology relying on techniques from the fields of software engineering, artificial intelligence, and knowledge representation. This paper contributes an aspect of this methodology, concentrating on the role of visualizations in CMEs. The authors state that visualization deserves a specialized theory that is based on music ontology and that is independent of the concrete, implemented graphical interface.