Multimedia Tasks Research Articles

Disentangling, Embedding and Ranking Label Cues for Multi-Label Image Recognition

Multi-label image recognition is a fundamental but challenging computer vision and multimedia task. Great progress has been achieved by exploiting label correlations among these multiple labels associated with a single image, which is the most crucial issue for multi-label image recognition. In this paper, to explicitly model label correlations, we propose a unified deep learning framework to Disentangle, Embed and Rank (DER) the corresponding label cues. Specifically, we first obtain class-aware disentangled maps (CADMs) by reforming deep activations in accordance with the class-specific recognition weights. Then, after transforming CADMs into the corresponding label vectors, we propose an embedding operation from a metric learning perspective to pull the relevant label vectors together and push irrelevant label vectors away. Furthermore, a ranking operation is employed, which aims to accurately and robustly measure the similarity/dissimilarity of these label vectors. Our model can be trained in an end-to-end manner with only image-level supervision, during which the proposed embedding and ranking operations can contribute to the CADMs learning through back-propagation. In addition, the obtained CADMs are aggregated and further used as an essential feature stream for the final multi-label classification. We conduct extensive experiments on three commonly used multi-label benchmark datasets. Quantitative results show that our model can significantly and consistently outperform previous competitive methods. Moreover, qualitative analysis of our DER proposal also reveals the effectiveness of our proposed model.

Dual Neural Networks Coupling Data Regression With Explicit Priors for Monocular 3D Face Reconstruction

We address the challenging issue of reconstructing a 3D face from one single image under various expressions and illuminations, which is widely applied in multimedia tasks. Methods built upon classical parametric morphable models (3DMMs) gain success on reconstructing the global geometry of a 3D face, but fail to precisely characterize local facial details. Recently, deep neural networks (DNN) have been applied to the reconstruction that directly predicts depth maps, showing compelling performance on detail recovery. Unfortunately, their reconstruction is prone to structural distortions owing to the lack of explicit prior constraints. In this paper, we propose dual neural networks that optimize one energy coupling data fitting with local explicit geometric prior. Specifically, we build one residual network upon traditional convolution layers in order to directly predict 3D structures by fitting an input image. Meanwhile, we devise a novel architecture stacking shallow networks to refine 3D clouds with geometric priors given by Markov random fields (MRFs). Quantitative evaluations demonstrate the superior performance of the dual networks over either end-to-end DNNs or parametric models. Comparisons with the state-of-the-art also show competitive reconstruction quality on various conditions.

Channel Compression: Rethinking Information Redundancy Among Channels in CNN Architecture

Model compression and acceleration are attracting increasing attentions due to the demand for embedded devices and mobile applications. Research on efficient convolutional neural networks (CNNs) aims at removing feature redundancy by decomposing or optimizing the convolutional calculation. In this work, feature redundancy is assumed to exist among channels in CNN architectures, which provides some leeway to boost calculation efficiency. Aiming at channel compression, a novel convolutional construction named compact convolution is proposed to embrace the progress in spatial convolution, channel grouping and pooling operation. Specifically, the depth-wise separable convolution and the point-wise interchannel operation are utilized to efficiently extract features. Different from the existing channel compression method which usually introduces considerable learnable weights, the proposed compact convolution can reduce feature redundancy with no extra parameters. With the point-wise interchannel operation, compact convolutions implicitly squeeze the channel dimension of feature maps. To explore the rules on reducing channel redundancy in neural networks, the comparison is made among different point-wise interchannel operations. Moreover, compact convolutions are extended to tackle with multiple tasks, such as acoustic scene classification, sound event detection and image classification. The extensive experiments demonstrate that our compact convolution not only exhibits high effectiveness in several multimedia tasks, but also can be efficiently implemented by benefiting from parallel computation.

Memory Augmented Deep Recurrent Neural Network for Video Question Answering.

Video question answering (VideoQA) is a very important but challenging multimedia task, which automatically analyzes questions and videos and generates accurate answers. However, research on VideoQA is still in its infancy. In this article, we propose a novel memory augmented deep recurrent neural network (MA-DRNN) model for VideoQA, which features a new method for encoding videos and questions, and memory augmentation using the emerging differentiable neural computer (DNC). Specifically, we encode textual (questions) information before visual (videos) information, which leads to better visual-textual representations. Moreover, we leverage DNC (with an external memory) for storing and retrieving useful information in questions and videos, and modeling the long-term visual-textual dependence. To evaluate the proposed model, we conducted extensive experiments using the VTW data set and MSVD-QA data set, which are both Widely used large-scale video data sets for language-level understanding. The experimental results have well validated the proposed model and showed that it outperforms the state-of-the-art in terms of various accuracy-related metrics.

Pay Attention to the Activations: A Modular Attention Mechanism for Fine-Grained Image Recognition

Fine-grained image recognition is central to many multimedia tasks such as search, retrieval and captioning. Unfortunately, these tasks are still challenging since the appearance of samples of the same class can be more different than those from different classes. Attention has been typically implemented in neural networks by selecting the most informative regions of the image that improve classification. In contrast, in this paper, attention is not applied at the image level but to the convolutional feature activations. In essence, with our approach, the neural model learns to attend to lower-level feature activations without requiring part annotations and uses those activations to update and rectify the output likelihood distribution. The proposed mechanism is modular, architecture-independent and efficient in terms of both parameters and computation required. Experiments demonstrate that well-known networks such as Wide Residual Networks and ResNeXt, when augmented with our approach, systematically improve their classification accuracy and become more robust to changes in deformation and pose and to the presence of clutter. As a result, our proposal reaches state-of-the-art classification accuracies in CIFAR-10, the Adience gender recognition task, Stanford Dogs, and UEC-Food100 while obtaining competitive performance in ImageNet, CIFAR-100, CUB200 Birds, and Stanford Cars. In addition, we analyze the different components of our model, showing that the proposed attention modules succeed in finding the most discriminative regions of the image. Finally, as a proof of concept, we demonstrate that with only local predictions, an augmented neural network can successfully classify an image before reaching any fully connected layer, thus reducing the computational amount up to 10%.

Multimedia intelligent fog computing scheme based on robust perception for complex networks

The dynamic topology of complex networks, the randomness of environmental noise and the uncertainty of data links make it difficult to guarantee the quality of multimedia services. This paper combines synchronization control, robustness perception and fog computing in complex networks to improve the quality of multimedia services, and designs a series of architecture and control algorithms for complex networks. Firstly, a synchronization control algorithm for complex networks is proposed based on the analysis of the multi-edge data flow cannot be exhausted, the mixing of multi-direction information transmission delay, the undirected multi-period characteristics of information collection delay and the requirements of multi-media tasks with different attributes. Then, aiming at optimizing the extraction of multimedia desired quality in complex environment, a multimedia transmission control algorithm is designed based on the robustness of complex network. This algorithm can accurately extract the desired multimedia signal from the environment noise, video, image and other redundant multimedia signals. Secondly, by adjusting the minimum spanning tree size of multipath system and scheduling multimedia data transmission, an intelligent algorithm for multimedia fog computation is designed. Finally, mathematical analysis and simulation experiments verify the correctness of the proposed algorithm from the aspects of real-time, feasibility, execution efficiency and resource utilization.

Electronic Educational Environment Moodle in English Language Training

The article deals with the causes, methods and functions of using E-learning in the process of training students English, as well as changing teacher’s and learner’s roles in the educational process. The introduction of information and communication technologies in educational and research processes is a compulsory requirement to implement the concept of modernization in Russian education. Recently, interest in English as a means of international communication has increased significantly, it has already been recognized as the language of professional communication in various fields of activity. The relevance of this work is undoubted. Modern higher school teachers are constantly looking for more effective approaches in training English and in the process of creating an electronic educational environment that would allow students access to various sources of information and help satisfy the interests of the modern student. The goal of the research is to increase the efficiency of training English in the system of higher professional education through the development and use of educational software products that apply modern information technology. Methods to achieve this goal include the method of using computer technology, methods of network planning and project management. The research showed that the organization of the information-instrumental educational environment uses a student centered approach, the development of an electronic educational and methodological complex (EEMC) is a search for an algorithm, most adapted to the student's cognitive needs, and EEMC in English creates the best conditions for the principle of interactivity. The practical significance of the study is determined by the fact that its materials related to the method of creating multimedia tasks and virtual environments can be used in the practical work of universities, as well as in the system of further training for specialists.

Dithen: A Computation-as-a-Service Cloud Platform for Large-Scale Multimedia Processing

We present Dithen, a novel computation-as-a-service (CaaS) cloud platform specifically tailored to the parallel execution of large-scale multimedia tasks. Dithen handles the upload/download of both multimedia data and executable items, the assignment of compute units to multimedia workloads, and the reactive control of the available compute units to minimize the cloud infrastructure cost under deadline-abiding execution. Dithen combines three key properties: (i) the reactive assignment of individual multimedia tasks to available computing units according to availability and predetermined time-to-completion constraints; (ii) optimal resource estimation based on Kalman-filter estimates; (iii) the use of additive increase multiplicative decrease (AIMD) algorithms (famous for being the resource management in the transport control protocol) for the control of the number of units servicing workloads. The deployment of Dithen over Amazon EC2 spot instances is shown to be capable of processing more than 80,000 video transcoding, face detection and image processing tasks (equivalent to the processing of more than 116 GB of compressed data) for less than $1 in billing cost from EC2. Moreover, the proposed AIMD-based control mechanism, in conjunction with the Kalman estimates, is shown to provide for more than 27 percent reduction in EC2 spot instance cost against methods based on reactive resource estimation. Finally, Dithen is shown to offer a 38 to 500 percent reduction of the billing cost against the current state-of-the-art in CaaS platforms on Amazon EC2 (Amazon Lambda and Amazon Autoscale). A baseline version of Dithen is currently available at dithen.com under the “AutoScale” option.

Electronic Educational Environment Moodle in English Language Training

The article deals with the causes, methods and functions of using E-learning in the process of training students English, as well as changing teacher’s and learner’s roles in the educational process. The introduction of information and communication technologies in educational and research processes is a compulsory requirement to implement the concept of modernization in Russian education. Recently, interest in English as a means of international communication has increased significantly, it has already been recognized as the language of professional communication in various fields of activity. The relevance of this work is undoubted. Modern higher school teachers are constantly looking for more effective approaches in training English and in the process of creating an electronic educational environment that would allow students access to various sources of information and help satisfy the interests of the modern student. The goal of the research is to increase the efficiency of training English in the system of higher professional education through the development and use of educational software products that apply modern information technology. Methods to achieve this goal include the method of using computer technology, methods of network planning and project management. The research showed that the organization of the information-instrumental educational environment uses a student centered approach, the development of an electronic educational and methodological complex (EEMC) is a search for an algorithm, most adapted to the student's cognitive needs, and EEMC in English creates the best conditions for the principle of interactivity. The practical significance of the study is determined by the fact that its materials related to the method of creating multimedia tasks and virtual environments can be used in the practical work of universities, as well as in the system of further training for specialists.

TDCS: a new scheduling framework for real-time multimedia OS

ABSTRACTThe emerging real-time hyper-physical system (CPS), such as autonomous vehicle and live interactive media application, requires time deterministic behaviour. This is challenging to achieve by using the traditional general purpose operating system (GPOS). This paper presents a new design of the real-time operating system (OS) scheduling mechanism called ‘time deterministic cyclic scheduling’ (TDCS) mainly for live multimedia tasks processing. This new scheduler shares a similar philosophy as classic cyclic execution but with flexibility and dynamic configuration. This hybrid design is based on both time-reserved based cyclic execution and priority-based pre-emptive scheduling for mixed criticality applications. The simulation results show that this scheduling scheme can achieve predictable timing behaviour of task delay and jitter under high CPU utilisation. This shows that the proposed scheme is promising for low latency high-performance multimedia censoring tasks that occur in a periodic manner.

Priority-Based Cloud Computing Architecture for Multimedia-Enabled Heterogeneous Vehicular Users

In recent days, vehicles have been equipped with smart devices that offer various multimedia-related applications and services, such as smart driving assistance, traffic congestions, weather forecasting, road safety alarms, and many entertainment and comfort applications. Thus, these smart vehicles produce a large amount of multimedia-related data that require fast and real-time processing. However, due to constrained computing and storage capacities, such huge amounts of multimedia-related data cannot be processed in on-board standalone devices. Thus, multimedia cloud computing (MCC) has emerged as an economical and scalable computing technology that can process multimedia-related data efficiently while providing improved Quality of Service (QoS) to vehicular users from anywhere, at any time and on any device, at reduced costs. However, there are certain challenges, such as fast service response time and resource cost optimization, that can severely affect the performance of the MCC. Therefore, to tackle these issues, in this paper, we propose a dynamic priority-based architecture for the MCC. In the proposed scheme, we divide multimedia processing into four different subphases, while computing resources to each computing server are assigned dynamically, according to the workload, in order to process multimedia tasks according to the multimedia user Quality of Experience (QoE) requirements. The performance of the proposed scheme is evaluated in terms of service response time and resource cost optimization using the CloudSim simulator.

Multimedia Data Modelling Using Multidimensional Recurrent Neural Networks

Modelling the multimedia data such as text, images, or videos usually involves the analysis, prediction, or reconstruction of them. The recurrent neural network (RNN) is a powerful machine learning approach to modelling these data in a recursive way. As a variant, the long short-term memory (LSTM) extends the RNN with the ability to remember information for longer. Whilst one can increase the capacity of LSTM by widening or adding layers, additional parameters and runtime are usually required, which could make learning harder. We therefore propose a Tensor LSTM where the hidden states are tensorised as multidimensional arrays (tensors) and updated through a cross-layer convolution. As parameters are spatially shared within the tensor, we can efficiently widen the model without extra parameters by increasing the tensorised size; as deep computations of each time step are absorbed by temporal computations of the time series, we can implicitly deepen the model with little extra runtime by delaying the output. We show by experiments that our model is well-suited for various multimedia data modelling tasks, including text generation, text calculation, image classification, and video prediction.

Curvature determination in range images: new methods and comparison study

A study of methods that determine surface curvature in real and synthetic range images is considered here. Four new determination methods are introduced (two convolution-based and two spline-based). These new methods, as well as a number of existing determination methods, are comparatively evaluated in terms of their (1) accuracy and (2) computational performance (i.e., run time). Their behavior in two common but challenging multimedia tasks (surface rendering and object discrimination based in whole or part on curvature determination) is also considered. Our evaluations also include an analysis of (1) which methods are most suitable for application in particular scenarios and (2) behavior of the methods with respect to parameter settings. One of the considered use cases includes application to consumer-grade Kinect sensor data.

Exploring geometric information in CNN for image retrieval

Convolutional Neural Network (CNN) has brought significant improvements for various multimedia tasks. In contrast, image retrieval has not yet benefited as much since no training database is available. In this paper, we propose an unsupervised weighting scheme for pre-trained CNN models to adaptively emphasize image center. Different from the general preference for fully connected layers which represent abstract semantics, we aggregate the activations of convolutional layers on image patches to depict local patterns in details. It is an empirical observation that the target of searching is naturally the focus of an image. Thus we pooling the features with respect to their positions, since they innately maintain the geometric layout of an image. Experimental results on two benchmarks prove the effectiveness of our methods.

Unevenness measurement using the support vector machine and dynamic multiservice load balancing with modified genetic algorithm in cloud-based multimedia system

Cloud computing is the most concerned and growing field in the real world, which is used by various fields to handle and manage the multimedia application due to availability of more amount of resources. At the time of multiple multimedia requests entering into the cloud server, fining and provisioning the required resources for the incoming multimedia requests of different king would be a more difficult task. In the existing system centralised hierarchical cloud-based multimedia system (CMS) is introduced which consists of elements such as resource manager, cluster head and server clusters. This word can assign the user required resources effectively with reduced cost. Also, existing research overcomes the load balancing problem which might occur at the time of multiple multimedia services with different characteristics entering into the system and allocating them in the server without considering their load capacity level using genetic algorithm. However, genetic algorithm would fail to find the optimal resource with optimal load level due to local search optimisation problem. A, load balancing cannot be done effectively in case of arrival of multimedia tasks with varying characteristics. This problem is resolved in the proposed research methodology by introducing the novel load balancing system in which both task unevenness problems and the genetic algorithms local search optimisation problems are resolved. This paper proposed an effective multimedia load balancing (MLB) scheme for CMS using support vector machine (SVM) and dynamic multiservice load balancing with adaptive genetic algorithm (AGA) (MLB-SVM-AGA). In this work, SVM is introduced for the purpose of quantifying the unevenness in the utilisation of multiple resources on a resource manager on the client side and confirmed at the server side in the each cluster. Unevenness scenario can be modelled as a mathematical hyperplane problem, but in most cases it is computationally intractable. Here, AGA is used for the purpose of solving the problem of dynamic load balancing. The experimental evaluation is conducted in the CloudSim toolkit for both proposed and existing research methodologies. The performance evaluation is done and the results demonstrate that AGA approach is able to dynamically spread the multimedia task load equally.

Resource scheduling approach for multimedia cloud content management

With the development of multimedia application and services, the multimedia technology has already permeated each aspect of our life. Multimedia cloud is used for processing multimedia services. However due to huge data volume, high concurrency, strict real-time, resource scheduling for content dissemination in multimedia cloud still remain challenges. In order to increase the user satisfaction and decrease completion time of content dissemination, the resource scheduling for content dissemination in multimedia cloud is proposed in this paper. The multimedia jobs are clustered according to user expectation and job complexity. The job with highest priority will be executed first. Moreover, considered multimedia task types and the impact of stragglers, the multimedia task scheduling based on task types and node workload is presented, which is a time-efficient scheduling approach. The experiments are conducted and the experiment results show that the job clustering algorithm-based user expectation and job complexity in multimedia cloud has better user satisfaction and shorter completion time, while the multimedia task scheduling based on task types and node workload can reduce completion time and achieve load-balancing.

Weakly-Supervised Deep Nonnegative Low-Rank Model for Social Image Tag Refinement and Assignment

It has been well known that the user-provided tags of social images are imperfect, i.e., there exist noisy, irrelevant or incomplete tags. It heavily degrades the performance of many multimedia tasks. To alleviate this problem, we propose a Weakly-supervised Deep Nonnegative Low-rank model (WDNL) to improve the quality of tags by integrating the low-rank model with deep feature learning. A nonnegative low-rank model is introduced to uncover the intrinsic relationships between images and tags by simultaneously removing noisy or irrelevant tags and complementing missing tags. The deep architecture is leveraged to seamlessly connect the visual content and the semantic tag. That is, the proposed model can well handle the scalability by assigning tags to new images. Extensive experiments conducted on two real-world datasets demonstrate the effectiveness of the proposed method compared with some state-of-the-art methods.

Revealing Event Saliency in Unconstrained Video Collection.

Recent progresses in multimedia event detection have enabled us to find videos about a predefined event from a large-scale video collection. Research towards more intrinsic unsupervised video understanding is an interesting but understudied field. Specifically, given a collection of videos sharing a common event of interest, the goal is to discover the salient fragments, i.e., the curt video fragments that can concisely portray the underlying event of interest, from each video. To explore this novel direction, this paper proposes an unsupervised event saliency revealing framework. It first extracts features from multiple modalities to represent each shot in the given video collection. Then, these shots are clustered to build the cluster-level event saliency revealing framework, which explores useful information cues (i.e., the intra-cluster prior, inter-cluster discriminability, and inter-cluster smoothness) by a concise optimization model. Compared with the existing methods, our approach could highlight the intrinsic stimulus of the unseen event within a video in an unsupervised fashion. Thus, it could potentially benefit to a wide range of multimedia tasks like video browsing, understanding, and search. To quantitatively verify the proposed method, we systematically compare the method to a number of baseline methods on the TRECVID benchmarks. Experimental results have demonstrated its effectiveness and efficiency.

ITI-CERTH participation to TRECVID 2011

This paper provides an overview of the tasks submitted to TRECVID 2011 by ITI-CERTH. ITICERTH participated in the Known-item search (KIS) as well as in the Semantic Indexing (SIN) and the Event Detection in Internet Multimedia (MED) tasks. In the SIN task, techniques are developed, which combine motion information with existing well-performing descriptors such as SURF, Random Forests and Bag-of-Words for shot representation. In the MED task, the trained concept detectors of the SIN task are used to represent video sources with model vector sequences, then a dimensionality reduction method is used to derive a discriminant subspace for recognizing events, and, nally, SVMbased event classiers are used to detect the underlying video events. The KIS search task is performed by employing VERGE, which is an interactive retrieval application combining retrieval functionalities in various modalities and exploiting implicit user feedback.

Ubiquitous interaction for ESP distance and blended learners

Purpose The purpose of this paper is to analyse some effective e-tools and interactive tasks to enhance language acquisition and competence building in today’s English for specific purposes (ESP) students in higher education programmes. Design/methodology/approach There are countless learning methods, often supported by the access of ubiquitous learning materials, which improve students’ engagement, group interaction and independent learning skills in foreign language learners. More frequently, students become successful, independent and cooperative creative learners by means of participating in multimodal virtual learning platforms that would help them to improve a variety of competences. This study is based on English applied to social work, a compulsory subject offered either in a blended (b-learning) or a distance (d-learning) learning environment as part of the degree in social work. It combines its correspondent course book with digital content and multimedia tasks. Findings Results suggest that a ubiquitous practice for ESP b/d learners can be innovative and successfully competence-addressed when implementing interactive learning tasks that can combine individual and group work inside and outside the classroom hours. Originality/value The challenge of this research is to integrate a ubiquitous learning approach so that students would interact more and, consequently, enhance their communicative language skills, both in b-learning and d-learning environments.