Multimedia Tasks Research Articles

Federated Deep Reinforcement Learning for Multimedia Task Offloading and Resource Allocation in MEC Networks

Federated Deep Reinforcement Learning for Multimedia Task Offloading and Resource Allocation in MEC Networks

Task-irrelevant decorative pictures increase cognitive load during text processing but have no effects on learning or working memory performance: an EEG and eye-tracking study

Decorative pictures (DP) are often used in multimedia task materials and are commonly considered so-called seductive details as they are commonly not task-relevant. Typically, DP result in mixed effects on behavioral performance measures. The current study focused on the effects of DP on the cognitive load during text reading and working memory task performance. The theta and alpha frequency band power of the electroencephalogram (EEG) and pupil dilation served as proxies of cognitive load. The number of fixations, mean fixation durations, and the number of transitions served as proxies of the attentional focus. For both, text reading and n-back working memory tasks, the presence and congruency of DP were manipulated in four task conditions. DP did neither affect behavioral performance nor subjective ratings of emotional–motivational factors. However, in both tasks, DP increased the cognitive load as revealed by the EEG alpha frequency band power and (at least to some extent) by subjective effort ratings. Notably, the EEG alpha frequency band power was a quite reliable and sensitive proxy of cognitive load. Analyzing the EEG data stimulus-locked and fixation-related, the EEG alpha frequency band power revealed a difference in global and local cognitive load. In sum, the current study underlines the feasibility and use of EEG for multimedia research, especially when combined with eye-tracking.

Multimedia Computation Offloading for 5G Networks Using the Volcano Eruption Optimization Algorithm

The communication services of multimedia, such as live online and short videos, gained significant attention and have been prominent in daily social interactions due to the fast development of 5G technology. However, the increase in multimedia communication requirements presents further difficulties for both the communication capacity of the wireless network and its processing capability. Mobile Edge Computing (MEC) is considered an effective technology to overcome the mentioned challenges. The subject of long-term task allocation including resource coordination is the primary focus of this study, with a special emphasis on multimedia services that must be processed, uploaded, and shared in network. The optimization issue is formulated as a stochastic optimization issue to reduce the system’s time-average energy consumption. An online energy-efficient task allocation and computing offloading method is proposed to determine task allocation, coordinate and optimize wireless, computing resource allocation, considering dynamic wireless state and service delay constraints. The proposed method is implemented in MATLAB and the efficacy of the MCO-VEA framework is estimated and performance metrics, such as minimum user rate, sum rate, energy, and spectral performance are analyzed. Then, the performance of MEC-MCO-VEA provides 22.35%, 25.84% and 36.58% higher throughput and 25.45%, 41.28%, and 28.56% higher energy efficiency compared to the existing models, such as an energy-effective multimedia task assignment with computing offloading for mobile edge computing (BA-LO-EE), energy efficiency using a communication deep neural network optimized with power allocation optimization (EE-CDNN-PAO), and energy efficiency using deep transfer deterministic policy gradient with deep deterministic policy gradient algorithm (DT-DPG-EE), respectively.

Enhancing Multimodal Information Retrieval Through Integrating Data Mining and Deep Learning Techniques

Multimodal information retrieval, the task of re trieving relevant information from heterogeneous data sources such as text, images, and videos, has gained significant attention in recent years due to the proliferation of multimedia content on the internet. This paper proposes an approach to enhance multimodal information retrieval by integrating data mining and deep learning techniques. Traditional information retrieval systems often struggle to effectively handle multimodal data due to the inherent complexity and diversity of such data sources. In this study, we leverage data mining techniques to preprocess and structure multimodal data efficiently. Data mining methods enable us to extract valuable patterns, relationships, and features from different modalities, providing a solid foundation for sub- sequent retrieval tasks. To further enhance the performance of multimodal information retrieval, deep learning techniques are employed. Deep neural networks have demonstrated their effectiveness in various multimedia tasks, including image recognition, natural language processing, and video analysis. By integrating deep learning models into our retrieval framework, we aim to capture complex intermodal dependencies and semantically rich representations, enabling more accurate and context-aware retrieval.

Improving the Performance of Delay-sensitive Multimedia Computation in Automotive Networks with DP-ERACOM using CloudSim Simulator

Abstract: We propose a Dynamic Priority-based Efficient Resource Allocation and Computing (DP-ERACOM) scheme for processing delay-sensitive multimedia computation in automotive networks. The proposed scheme breaks down each multimedia task into four smaller tasks and dynamically distributes resources based on the priority of the multimedia tasks. The three main processing and computing units in the proposed scheme are the load manager, compute cluster unit, and transmission unit. We evaluate the performance of the proposed scheme using CloudSim simulator. The results show that the proposed scheme can significantly improve the performance of delay-sensitive multimedia computation in automotive networks.

Flood-Related Multimedia Benchmark Evaluation: Challenges, Results and a Novel GNN Approach.

This paper discusses the importance of detecting breaking events in real time to help emergency response workers, and how social media can be used to process large amounts of data quickly. Most event detection techniques have focused on either images or text, but combining the two can improve performance. The authors present lessons learned from the Flood-related multimedia task in MediaEval2020, provide a dataset for reproducibility, and propose a new multimodal fusion method that uses Graph Neural Networks to combine image, text, and time information. Their method outperforms state-of-the-art approaches and can handle low-sample labelled data.

A Boundary-Aware Network for Shadow Removal

Shadow removal is a challenging computer vision and multimedia task that aims to restore image content in shadow regions. The state-of-the-art shadow removal methods introduce artifacts near shadow boundaries or inconsistencies between shadow and nonshadow areas, which can be easily noticed by the human eye at first glance. In this paper, we design a boundary-aware shadow removal network (BA-ShadowNet) that improves shadow removal accuracy by increasing the removal performance at shadow boundaries. In contrast with previously developed methods, which usually consider shadow boundary optimization to be a postprocessing technique, our method performs shadow removal and shadow boundary optimization simultaneously. For this purpose, the proposed BA-ShadowNet is designed as a multiscale encoder-decoder structure, where the decoder consists of a shadow removal branch and a shadow optimization branch. An interaction module is then introduced to fuse and exchange the features of the two branches. This module facilitates the removal branch in perceiving the locations and colors of shadow boundaries. Additionally, it optimizes the boundary branch according to the image context extracted from the removal branch. A three-term loss function is further developed to supervise the shadow removal results and to address the issue of imbalanced supervision between shadow boundary pixels and pixels inside shadows. Extensive experiments conducted on the ISTD+ and SRD datasets demonstrate that the proposed BA-ShadowNet greatly outperforms the state-of-the-art methods with respect to shadow removal.

Knowing What it is: Semantic-Enhanced Dual Attention Transformer

Attention has become an indispensable component of the models of various multimedia tasks like Image Captioning (IC) and Visual Question Answering (VQA). However, most existing attention modules are designed for capturing the spatial dependency, and are still insufficient in semantic understanding, e.g.}, the categories of objects and their attributes, which is also critical for image captioning. To compensate for this defect, we propose a novel attention module termed Channel-wise Attention Block (CAB) to model channel-wise dependency for both visual modality and linguistic modality, thereby improving semantic learning and multi-modal reasoning simultaneously. Specifically, CAB has two novel designs to tickle with the high overhead of channel-wise attention, which are the reduction-reconstruction block structure and the gating-based attention prediction. Based on CAB, we further propose a novel Semantic-enhanced Dual Attention Transformer (termed SDATR), which combines the merits of spatial and channel-wise attentions. To validate SDATR, we conduct extensive experiments on the MS COCO dataset and yield new state-of-the-art performance of 134.5 CIDEr score on COCO Karpathy test split and 136.0 CIDEr score on the official online testing server. To examine the generalization of SDATR, we also apply it to the task of visual question answering, where superior performance gains are also witnessed.

Emotional Semantics-Preserved and Feature-Aligned CycleGAN for Visual Emotion Adaptation

Thanks to large-scale labeled training data, deep neural networks (DNNs) have obtained remarkable success in many vision and multimedia tasks. However, because of the presence of domain shift, the learned knowledge of the well-trained DNNs cannot be well generalized to new domains or datasets that have few labels. Unsupervised domain adaptation (UDA) studies the problem of transferring models trained on one labeled source domain to another unlabeled target domain. In this article, we focus on UDA in visual emotion analysis for both emotion distribution learning and dominant emotion classification. Specifically, we design a novel end-to-end cycle-consistent adversarial model, called CycleEmotionGAN++. First, we generate an adapted domain to align the source and target domains on the pixel level by improving CycleGAN with a multiscale structured cycle-consistency loss. During the image translation, we propose a dynamic emotional semantic consistency loss to preserve the emotion labels of the source images. Second, we train a transferable task classifier on the adapted domain with feature-level alignment between the adapted and target domains. We conduct extensive UDA experiments on the Flickr-LDL and Twitter-LDL datasets for distribution learning and ArtPhoto and Flickr and Instagram datasets for emotion classification. The results demonstrate the significant improvements yielded by the proposed CycleEmotionGAN++ compared to state-of-the-art UDA approaches.

Rethinking prediction alignment in one-stage object detection

Owing to their excellent performance and efficiency, one-stage detectors have been widely used in multimedia tasks, such as temporal action detection, object tracking, and video detection. However, misalignment between classification and regression branches limits the accuracy of the detector. Most existing works add an auxiliary branch or adopt a specific sample assignment strategy to alleviate this problem, but with little effect. In this paper, we attribute this to incomplete branch interactions and propose a comprehensive Predictive Aligned Object Detector (PAOD), which can better correlate two subtasks. Specifically, our proposed PAOD achieves a better trade-off between prediction-interactive and prediction-specific by adopting an Iterative Aggregation Module (IAM) and a Mutual Constraint Module (MCM). We also design an aligned label assignment with an adaptive metric and re-weighting mechanism to further narrow the misalignment between prediction heads. With negligible additional overhead, PAOD achieves 50.4 AP at single-model single-scale testing on the MS-COCO branch, which demonstrates the effectiveness of our proposal. Notably, PAOD consistently outperforms previous sota such as ATSS (47.7 AP), BorderDet (48.0 AP) and GFL (48.2 AP) by a large margin on COCO test-dev dataset, and achieves better performance than various dense detectors on Pascal VOC and CrowdHuman datasets. Code is available at https://github.com/JunruiXiao/PAOD.

Scalable Discrete Matrix Factorization and Semantic Autoencoder for Cross-Media Retrieval.

Hashing methods have sparked great attention on multimedia tasks due to their effectiveness and efficiency. However, most existing methods generate binary codes by relaxing the binary constraints, which may cause large quantization error. In addition, most supervised cross-modal approaches preserve the similarity relationship by constructing an n×n large-size similarity matrix, which requires huge computation, making these methods unscalable. To address the above challenges, this article presents a novel algorithm, called scalable discrete matrix factorization and semantic autoencoder method (SDMSA). SDMSA is a two-stage method. In the first stage, the matrix factorization scheme is utilized to learn the latent semantic information, the label matrix is incorporated into the loss function instead of the similarity matrix. Thereafter, the binary codes can be generated by the latent representations. During optimization, we can avoid manipulating a large n×n similarity matrix, and the hash codes can be generated directly. In the second stage, a novel hash function learning scheme based on the autoencoder is proposed. The encoder-decoder paradigm aims to learn projections, the feature vectors are projected to code vectors by encoder, and the code vectors are projected back to the original feature vectors by the decoder. The encoder-decoder scheme ensures the embedding can well preserve both the semantic and feature information. Specifically, two algorithms SDMSA-lin and SDMSA-ker are developed under the SDMSA framework. Owing to the merit of SDMSA, we can get more semantically meaningful binary hash codes. Extensive experiments on several databases show that SDMSA-lin and SDMSA-ker achieve promising performance.

Improving projected fuzzy K-means clustering via robust learning

Fuzzy K-Means clustering has been an attractive research area for many multimedia tasks. Due to the interference of the noise and outliers, the performance of fuzzy K-Means clustering has been limited. In this paper, a projected fuzzy K-Means clustering method, referred to as Robust Projected Fuzzy K-Means (RPFKM) is proposed as a response to such a challenge. RPFKM improves fuzzy K-Means clustering in three perspectives. First, unlike existing fuzzy K-Means algorithms where the clustering process is conducted in the original space, RPFKM learns the fuzzy membership relationship between samples and prototypes in the low-dimensional space to eliminate the influence of the noise and irrelevant features. Second, it employs the ℓ21 norm to reduce the contribution of outliers to the learning of prototypes. Third, it also considers the sensitivity of fuzzy clustering to the number of reduced dimensions, and the reconstruction term is introduced to hold the main energy of the original data. Furthermore, an iterative re-weighted algorithm is developed to solve the proposed method. The evaluation results of our proposed method and the state-of-the-art methods on real-world and synthetic data sets show the effectiveness and efficiency of our approach.

Knowledge-Driven Generative Adversarial Network for Text-to-Image Synthesis

Text-to-Image (T2I) synthesis is a challenging task that aims to convert natural language descriptions to real images. It remains an open problem mainly due to the diversity of text descriptions, which poses a huge obstacle in generating vivid and relevant images. Moreover, the existing evaluation metrics in T2I synthesis are mainly used to evaluate the visual quality of the generated images, while the semantic consistency between the two modalities is often ignored. To address these issues, we present a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Knowledge-Driven Generative Adversarial Network</i> , termed KD-GAN, and a new evaluation system, named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Pseudo Turing Test</i> (PTT for short). Concretely, KD-GAN takes a further step in imitating the behavior of human painting, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , drawing an image according to reference knowledge. The introduction of reference knowledge in KD-GAN not only improves the quality of the generated images but also enhances the semantic consistency between them and the input texts. In addition, KD-GAN can also greatly avoid some flaws against common sense during image generation, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , skiing in the blue sky. The proposed PTT is an important supplement to the existing evaluation system of T2I synthesis. It includes a set of pseudo-experts of different multimedia tasks to evaluate the semantic consistency between the given texts and the generated images. To validate the proposed KD-GAN, we conducted extensive experiments on two benchmark datasets, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , Caltech-UCSD Birds (CUB), and MS-COCO (COCO). The experimental results demonstrate that KD-GAN outperforms state-of-the-art methods on IS, FID, and the proposed PTT metrics. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref> <fn id="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><label>¹</label> The codes of KD-GAN are at [Online]. Available: <uri>https://github.com/pengjunn/KD-GAN</uri> and the codes and models of PTT are at [Online]. Available: <uri>https://github.com/pengjunn/PTT</uri>. </fn>

VRBagged-Net: Ensemble Based Deep Learning Model for Disaster Event Classification

A flood is an overflow of water that swamps dry land. The gravest effects of flooding are the loss of human life and economic losses. An early warning of these events can be very effective in minimizing the losses. Social media websites such as Twitter and Facebook are quite effective in the efficient dissemination of information pertinent to any emergency. Users on these social networking sites share both textual and rich content images and videos. The Multimedia Evaluation Benchmark (MediaEval) offers challenges in the form of shared tasks to develop and evaluate new algorithms, approaches and technologies for explorations and exploitations of multimedia in decision making for real time problems. Since 2015, the MediaEval has been running a shared task of predicting several aspects of flooding and through these shared tasks, many improvements have been observed. In this paper, the classification framework VRBagged-Net is proposed and implemented for flood classification. The framework utilizes the deep learning models Visual Geometry Group (VGG) and Residual Network (ResNet), along with the technique of Bootstrap aggregating (Bagging). Various disaster-based datasets were selected for the validation of the VRBagged-Net framework. All the datasets belong to the MediaEval Benchmark Workshop, this includes Disaster Image Retrieval from Social Media (DIRSM), Flood Classification for Social Multimedia (FCSM) and Image based News Topic Disambiguation (INTD). VRBagged-Net performed encouraging well in all these datasets with slightly different but relevant tasks. It produces Mean Average Precision at different levels of 98.12, and Average Precision at 480 of 93.64 on DIRSM. On the FCSM dataset, it produces an F1 score of 90.58. Moreover, the framework has been applied on the dataset of Image-Based News Topic Disambiguation (INTD), and exceeds the previous best result by producing an F1 evaluation of 93.76. The VRBagged-Net with a slight modification also ranked first in the flood-related Multimedia Task at the MediaEval Workshop 2020.

A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging.

Gastrointestinal (GI) endoscopy has been an active field of research motivated by the large number of highly lethal GI cancers. Early GI cancer precursors are often missed during the endoscopic surveillance. The high missed rate of such abnormalities during endoscopy is thus a critical bottleneck. Lack of attentiveness due to tiring procedures, and requirement of training are few contributing factors. An automatic GI disease classification system can help reduce such risks by flagging suspicious frames and lesions. GI endoscopy consists of several multi-organ surveillance, therefore, there is need to develop methods that can generalize to various endoscopic findings. In this realm, we present a comprehensive analysis of the Medico GI challenges: Medical Multimedia Task at MediaEval 2017, Medico Multimedia Task at MediaEval 2018, and BioMedia ACM MM Grand Challenge 2019. These challenges are initiative to set-up a benchmark for different computer vision methods applied to the multi-class endoscopic images and promote to build new approaches that could reliably be used in clinics. We report the performance of 21 participating teams over a period of three consecutive years and provide a detailed analysis of the methods used by the participants, highlighting the challenges and shortcomings of the current approaches and dissect their credibility for the use in clinical settings. Our analysis revealed that the participants achieved an improvement on maximum Mathew correlation coefficient (MCC) from 82.68% in 2017 to 93.98% in 2018 and 95.20% in 2019 challenges, and a significant increase in computational speed over consecutive years.

Spatio-Spectral Feature Fusion for Low-Light Image Enhancement

Low-light image enhancement aims to improve an image's visual quality, which is essential for many downstream computer vision and multimedia tasks. Existing spatial-domain low-light enhancement methods barely focus on the regions containing object boundaries, which take the most informative characteristics. However, solely focusing on enhancing high-frequency details not only causes over-sharpening of an image but also leads to color distortion. In this paper, we propose a novel spatio-spectral feature fusion network (S2F2N), that involves a frequency-feature representation branch (FRB) and a spatial-feature representation branch (SRB) to learn the domain-specific representation individually. Moreover, a spatial-channel mixed attention block (MAB) is introduced to learn the joint representation of spatio-spectral features for final image relighting. Extensive experiments on several benchmark datasets demonstrate that our method can produce high fidelity results for low-light images.

Bayesian Graph Convolutional Neural Networks via Tempered MCMC

Deep learning models, such as convolutional neural networks, have long been applied to image and multi-media tasks, particularly those with structured data. More recently, there has been more attention to unstructured data that can be represented via graphs. These types of data are often found in health and medicine, social networks, and research data repositories. Graph convolutional neural networks have recently gained attention in the field of deep learning that takes advantage of graph-based data representation with automatic feature extraction via convolutions. Given the popularity of these methods in a wide range of applications, robust uncertainty quantification is vital. This remains a challenge for large models and unstructured datasets. Bayesian inference provides a principled approach to uncertainty quantification of model parameters for deep learning models. Although Bayesian inference has been used extensively elsewhere, its application to deep learning remains limited due to the computational requirements of the Markov Chain Monte Carlo (MCMC) methods. Recent advances in parallel computing and advanced proposal schemes in MCMC sampling methods has opened the path for Bayesian deep learning. In this paper, we present Bayesian graph convolutional neural networks that employ tempered MCMC sampling with Langevin-gradient proposal distribution implemented via parallel computing. Our results show that the proposed method can provide accuracy similar to advanced optimisers while providing uncertainty quantification for key benchmark problems.

ТВОРЧЕСКИЕ ЗАДАНИЯ ПО ЛИТЕРАТУРЕ: ОТ ИСТОРИИ К СОВРЕМЕННОСТИ, ОТ КОНСЕРВАЦИИ ФОРМ К ИХ РАЗВИТИЮ И ТРАНСФОРМАЦИИ (ОБЩИЙ ОБЗОР)

Introduction to the paper raises the questions: what are the causes of today’s crisis faced by the “traditional,” which has lost its creative character, and if there are ways out of this crisis. Materials and methods. When searching for the answers to these questions, the study relies on the comparative analysis of the sources and shows how Russian teaching methodology experts of the 20th century solved the “essay problem,” what tasks were offered for the text written by students to be an original “answer” in their dialogue with the literary text. Bibliography overview. A whole host of the sources used impart the character of an analytical review to this part of the paper. It aims to convince the reader that the focus on “creative reading” (Abakumov) and “creative writing” (Abakumov) was one of the leading trends in the methodology of the 20th century. This section states that the literary creativity of students nowadays can and should be supported with digital tools. Digitalization of modern literary education, including the application of new creative tasks in the educational process, is a condition for its development in the 21st century. Results. This part of the paper presents possible multimedia creative tasks (podcast, book trailer, longread, video poetry, and others) and indicates that the latter performed using IT tools are not a rejection of conventional creative tasks but rather their development and transformation in the digital environment. The proposed thesis is illustrated by referring to fanfiction and showing what traditional creative task forms preceded fanfiction. These tasks (as opposed to writing a traditional essay) are shown to become valuable and meaningful for the students because they are characterized by convergence and participativeness. It is through these properties of the task’s students are involved in the participatory culture within which the texts of fiction live today. This section concludes that the proposed system of creative multimedia tasks should further be discussed, experimentally tested, and adjusted based on the testing results. In conclusion, according to the teaching methodology experts of the 20th century, who talked about student creativity as a necessary condition for understanding a literary text and a productive dialogue with it, the methodological science objectives to be set concerning the issue presented in the title of the paper are identified. These include: creating a glossary of terms that defines possible creative tasks of a multimedia nature and generating methodological recommendations that describe an algorithm for their development and the technology for including them in the educational process. Keywords: traditional and multimedia creative tasks, value and meaning, convergence and participation, fanfiction, classification of multimedia creative tasks

Story-driven Video Editing

This paper proposes a novel multimedia task: story-driven video editing. Given a story paragraph, this task aims to retrieve related video segments from a gallery of collected video segments and compose them into a video sequence by the storyline order. Our proposed baseline solution consists of three modules: a retrieval module, which returns lists of candidate segments for all query sentences in the story paragraph using an object-aware sentence-segment matching method; a sequence candidate proposal module, which aggregates the retrieved segment sets into a sequence proposal by the submodular optimization method; a sorting module, which arranges the candidates according to the storyline of the paragraph using the Sinkhorn network. We build a benchmark for this task including a reorganized version of the ActivityNet Captions dataset, a well-defined quantitative metric called Evaluation of Segment-to-Sequence Matching (ESSM) for measuring the difference between the generated video segment sequence and the ground truth. Quantitative results of the proposed baseline solution are reported. We hope this new task and benchmark will bring broad research attention and push forward a lot of novel online short video editing applications.

Using eye-tracking and EEG to study the mental processing demands during learning of text-picture combinations

Using and combining eye-tracking and EEG frequency band power as process measures, in the current study we were interested in the mental processing demands during learning of text-picture combinations that either enabled or prohibited text-picture integration (TPI). In the mismatch condition, the textual and pictorial information being dissimilar, TPI was not possible, whereas in the match and the partial-match condition, the textual and pictorial information being identical respective complementary, TPI was possible. We expected mental processing demands to be higher in the mismatch condition, when pictorial and textual information had to be processed and memorized as separate representations, compared to the match and partial-match conditions when TPI was possible. As expected, on virtually all process measures we observed increased mental processing demands when two mental representations had to be processed and memorized compared to the two conditions where TPI was possible. The EEG alpha and theta frequency band power data corroborated and extended the eye-tracking measures of mental processing demands. In addition, we performed a fixation-related EEG frequency band power analysis that also corroborated the results of the classic stimulus-locked EEG frequency band power analysis, exemplifying the use of this former methodology in the context of complex multimedia task materials.