Multimedia Analytics Research Articles

Real-time Multimedia Analytics for IoT Applications: Leveraging Machine Learning for Insights

Real-time Multimedia Analytics for IoT Applications: Leveraging Machine Learning for Insights

Memorability-based multimedia analytics for robotic interestingness prediction system using trimmed Q-learning algorithm.

Mobile robots are increasingly employed in today's environment. Perceiving the environment to perform a task plays a major role in the robots. The service robots are wisely employed in the fully (or) partially known user's environment. The exploration and exploitation of the unknown environment is a tedious task. This paper introduces a novel Trimmed Q-learning algorithm to predict interesting scenes via efficient memorability-oriented robotic behavioral scene activity training. The training process involves three stages: online learning and short-term and long-term learning modules. It is helpful for autonomous exploration and making wiser decisions about the environment. A simplified three-stage learning framework is introduced to train and predict interesting scenes using memorability. A proficient visual memory schema (VMS) is designed to tune the learning parameters. A role-based profile arrangement is made to explore the unknown environment for a long-term learning process. The online and short-term learning frameworks are designed using a novel Trimmed Q-learning algorithm. The underestimated bias in robotic actions must be minimized by introducing a refined set of practical candidate actions. Finally, the recalling ability of each learning module is estimated to predict the interesting scenes. Experiments conducted on public datasets, SubT, and SUN databases demonstrate the proposed technique's efficacy. The proposed framework has yielded better memorability scores in short-term and online learning at 72.84% and in long-term learning at 68.63%.

Implementation of a Multimodal Knowledge-Exchange Platform to Provide Trauma Critical Care Education During the Ongoing Conflict in Ukraine

The conflict in Ukraine has forced civilian hospitals with limited trauma and battlefield medicine experience to care for casualties of war, placing significant strain on the health care system. Using the Checklist for Early Recognition and Treatment of Acute Illness and Injury (CERTAIN) program, a multimodal trauma critical care knowledge-exchange platform was created for clinicians practicing in these institutions. To describe the development and implementation of the CERTAIN for Ukraine program and to evaluate the reach of this intervention, together with participant engagement and satisfaction. This quality improvement study included clinicians caring for critically ill patients during the ongoing Ukrainian conflict who were part of a community developed using a messaging app. The program was implemented by a group of international trauma and critical care experts in collaboration with critical care leaders from the Shupyk National Healthcare University in Kyiv, Ukraine. This study evaluates data collected from the CERTAIN for Ukraine program from its launch on April 9, 2022, to August 31, 2022. The initiative comprised a longitudinal series of interactive tele-education sessions, a webpage containing the CERTAIN approach and current trauma critical care guidelines translated into Ukrainian and Russian, and a private messaging chat for asynchronous discussion. Participant engagement and satisfaction were tracked using multimedia analytics and a post-session survey. Since program launch, 838 participants have joined the messaging group, and 6 tele-education sessions have been delivered, with 1835 total views. The CERTAIN website has had 3527 visits, mainly from Ukraine (1378 [39%]) and the United States (1060 [30%]). Of the 74 completed postsession surveys, 65 respondents (88%) rated the course content excellent or very good, and 73 (99%) recommended it to others. The findings of this quality improvement study indicate that, using widely available and low-cost platforms, knowledge was shared rapidly and efficiently to a large community of clinicians practicing in a wartime environment with broad-based engagement and a high level of learner satisfaction.

Imaging and Multimedia Analytics at the Edge 2023 Conference Overview and Papers Program

Imaging and Multimedia Analytics at the Edge 2023 Conference Overview and Papers Program

Artificial Intelligence (AI) and Machine Learning for Multimedia and Edge Information Processing

The advancements and progress in artificial intelligence (AI) and machine learning, and the numerous availabilities of mobile devices and Internet technologies together with the growing focus on multimedia data sources and information processing have led to the emergence of new paradigms for multimedia and edge AI information processing, particularly for urban and smart city environments. Compared to cloud information processing approaches where the data are collected and sent to a centralized server for information processing, the edge information processing paradigm distributes the tasks to multiple devices which are close to the data source. Edge information processing techniques and approaches are well suited to match current technologies for Internet of Things (IoT) and autonomous systems, although there are many challenges which remain to be addressed. The motivation of this paper was to survey these new paradigms for multimedia and edge information processing from several technological perspectives including: (1) multimedia analytics on the edge empowered by AI; (2) multimedia streaming on the intelligent edge; (3) multimedia edge caching and AI; (4) multimedia services for edge AI; and (5) hardware and devices for multimedia on edge intelligence. The review covers a wide spectrum of enabling technologies for AI and machine learning for multimedia and edge information processing.

Imaging and Multimedia Analytics at the Edge 2022 Conference Overview and Papers Program

We welcome research papers and presentations that explore advances in deep learning and imaging in a hybrid edge-cloud processing environment. Research topics include face detection and recognition; human detection and tracking; human behavior and action recognition, biometrics; gesture analysis; event detection; anomaly detection; object tracking and recognition; 3D and depth information for object recognition and reconstruction; pose estimation; multimedia and multimodal inference and understanding; content analysis, indexing, search, and retrieval.

STUDENTS’ PREFERENCES OF LEARNING MATERIALS DURING UNPRECEDENTED ONLINE LEARNING DUE TO COVID-19

Teaching and learning processes in higher education have been affected due to the COVID-19 pandemic. Face-to face learning has been transitioned to online learning. This study explored student preferences for online materials during unprecedented online learning due to COVID-19. Through an online questionnaire, quantitative data was collected from 104 students from the Department of Computer Science, Faculty of Defence and Science Technology. The factors focused on in analysing the student’s preferences in online learning material include the layout, and multimedia analytics approach. The results present how multimedia analytics such as video, audio, graphic, and animation in online learning material play a significant role in the cognitive engagement and student’s motivation in online learning. Also, respondents preferred online learning materials with element of multimedia such as video and audio, rather than text only. This study presents suggestions on how to enhance online learning materials to improve online learning.

Machine learning for big multimedia analytics

Machine learning for big multimedia analytics

II-20: Intelligent and pragmatic analytic categorization of image collections.

In this paper, we introduce 11-20 (Image Insight 2020), a multimedia analytics approach for analytic categorization of image collections. Advanced visualizations for image collections exist, but they need tight integration with a machine model to support the task of analytic categorization. Directly employing computer vision and interactive learning techniques gravitates towards search. Analytic categorization, however, is not machine classification (the difference between the two is called the pragmatic gap): a human adds/redefines/deletes categories of relevance on the fly to build insight, whereas the machine classifier is rigid and non-adaptive. Analytic categorization that truly brings the user to insight requires a flexible machine model that allows dynamic sliding on the exploration-search axis, as well as semantic interactions: a human thinks about image data mostly in semantic terms. 11-20 brings three major contributions to multimedia analytics on image collections and towards closing the pragmatic gap. Firstly, a new machine model that closely follows the user's interactions and dynamically models her categories of relevance. II-20's machine model, in addition to matching and exceeding the state of the art's ability to produce relevant suggestions, allows the user to dynamically slide on the exploration-search axis without any additional input from her side. Secondly, the dynamic, 1-image-at-a-time Tetris metaphor that synergizes with the model. It allows a well-trained model to analyze the collection by itself with minimal interaction from the user and complements the classic grid metaphor. Thirdly, the fast-forward interaction, allowing the user to harness the model to quickly expand ("fast-forward") the categories of relevance, expands the multimedia analytics semantic interaction dictionary. Automated experiments show that II-20's machine model outperforms the existing state of the art and also demonstrate the Tetris metaphor's analytic quality. User studies further confirm that II-20 is an intuitive, efficient, and effective multimedia analytics tool.

Multi-source and heterogeneous multimedia analytics

Multi-source and heterogeneous multimedia analytics

Industrial defect detection by comparison with reference 3D CAD model

Fast track article for IS&T International Symposium on Electronic Imaging 2021: Imaging and Multimedia Analytics in a Web and Mobile World 2021 proceedings.

Visual document tracking and blockchain technologies in mobile world

Fast track article for IS&T International Symposium on Electronic Imaging 2021: Imaging and Multimedia Analytics in a Web and Mobile World 2021 proceedings.

An improved model for sentiment analysis on luxury hotel review

AbstractThis article proposes a heuristic model for sentiment analysis on luxury hotel reviews to analyse and explore marketing insights from attitudes and emotions expressed in reviews. We make several significant contributions to visual and multimedia analytics. This research will develop the practical application of visual and multimedia analytics as the research foundation is based on information analytics, geospatial analytics, statistical analytics and data management. Large amounts of data are generated by hotel customers on the Internet, which provides a good opportunity for managers and analysts to explore the hidden information. The analysis of luxury hotels involves different types of data, including real‐world scale data, high‐dimensional data and geospatial data. The diversity of data increases the difficulty of processing computational visual analytics. It leads to that some classical classification methods, which cost too much time and have high requirements for hardware, are excluded. The goal is to achieve a compromise between performance and cost. An experiment of this model is operated using data extracted from Booking.com. The entire framework of this experiment includes data collection, data preprocessing, feature engineering consisting of term frequency‐inverse document frequency and Doc2Vec based feature generation and feature selection, Random Forest classification, data analysis and data visualization. The whole process combines statistical analysis, review sentiment analysis and visual analysis to make full use of this dataset and gain more decision‐making information to improve luxury hotels' service quality. Compared with simple sentiment analysis, this integrated analytics in social media is expected to be used in practice to gain more insights. The result shows that luxury hotels should focus on staff training, cleanness of rooms and location choice to improve customer satisfaction. The sentiment distribution shows that scores are consistent with the emotion they show in reviews. Hotels in Spain have a much better average score than hotels in the other five countries. In the experiment, the sentiment analysis model is evaluated by receiver operating characteristic and precision‐recall curve. It is proved that this model performs well. Twenty most essential features have been listed for future adjustments to the model.

Social Multimedia Security and Suspicious Activity Detection in SDN using Hybrid Deep Learning Technique

Social multimedia traffic is growing exponentially with the increased usage and continuous development of services and applications based on multimedia. Quality of Service (QoS), Quality of Information (QoI), scalability, reliability and such factors that are essential for social multimedia networks are realized by secure data transmission. For delivering actionable and timely insights in order to meet the growing demands of the user, multimedia analytics is performed by means of a trust-based paradigm. Efficient management and control of the network is facilitated by limiting certain capabilities such as energy-aware networking and runtime security in Software Defined Networks. In social multimedia context, suspicious flow detection is performed by a hybrid deep learning based anomaly detection scheme in order to enhance the SDN reliability. The entire process is divided into two modules namely – Abnormal activities detection using support vector machine based on Gradient descent and improved restricted Boltzmann machine which facilitates the anomaly detection module, and satisfying the strict requirements of QoS like low latency and high bandwidth in SDN using end-to-end data delivery module. In social multimedia, data delivery and anomaly detection services are essential in order to improve the efficiency and effectiveness of the system. For this purpose, we use benchmark datasets as well as real time evaluation to experimentally evaluate the proposed scheme. Detection of malicious events like confidential data collection, profile cloning and identity theft are performed to analyze the performance of the system using CMU-based insider threat dataset for large scale analysis.

Human Fall Detection Based on Body Posture Spatio-Temporal Evolution.

Abnormal falls in public places have significant safety hazards and can easily lead to serious consequences, such as trampling by people. Vision-driven fall event detection has the huge advantage of being non-invasive. However, in actual scenes, the fall behavior is rich in diversity, resulting in strong instability in detection. Based on the study of the stability of human body dynamics, the article proposes a new model of human posture representation of fall behavior, called the “five-point inverted pendulum model”, and uses an improved two-branch multi-stage convolutional neural network (M-CNN) to extract and construct the inverted pendulum structure of human posture in real-world complex scenes. Furthermore, we consider the continuity of the fall event in time series, use multimedia analytics to observe the time series changes of human inverted pendulum structure, and construct a spatio-temporal evolution map of human posture movement. Finally, based on the integrated results of computer vision and multimedia analytics, we reveal the visual characteristics of the spatio-temporal evolution of human posture under the potentially unstable state, and explore two key features of human fall behavior: motion rotational energy and generalized force of motion. The experimental results in actual scenes show that the method has strong robustness, wide universality, and high detection accuracy.

Visual time period analysis: a multimedia analytics application for summarizing and analyzing eye-tracking experiments

Recently, an increasing need for sophisticated multimedia analytics tools has been observed, which is triggered by a rapid growth of multimedia collections and by an increasing number of scientific fields embedding images in their studies. Although temporal data is ubiquitous and crucial in many applications, such tools typically do not support the analysis of data along the temporal dimension, especially for time periods. An appropriate visualization and comparison of period data associated with multimedia collections would help users to infer new information from such collections. In this paper, we present a novel multimedia analytics application for summarizing and analyzing temporal data from eye-tracking experiments. The application combines three different visual approaches: Time∘diff, visual-information-seeking mantra, and multi-viewpoint. A qualitative evaluation with domain experts confirmed that our application helps decision makers to summarize and analyze multimedia collections containing period data.

Towards the Conceptual Retrieval of Multimedia Documentary: A Survey

Billions of active online users are continuously feeding the world with multimedia Big Data through their smart phones and PCs. These heterogenous productions are existing in different social media platforms, such as Facebook and Twitter, delivering a composite message in the form of audio, visual and textual signals. Analyzing multimedia Big Data to understand the intended delivered message, had been a challenge to audio, image, video and text processing researchers. Thanks to the recent advances in deep learning algorithms, researchers had been able to improve the performance of multimedia Big Data analytics and understanding techniques This paper presents a survey on how a multimedia file is analyzed, key challenges facing multimedia analysis, and how deep learning is helping conquer and advance beyond those challenges. Future directions of multimedia analysis are also addressed. The aim is to stay objective all through this study, bringing both empowering enhancements and in addition inescapable shortcomings, wishing to bring up fresh questions and stimulating new research frontiers for the reader.

Guest Editorial Trustworthiness in Social Multimedia Analytics and Delivery

The papers in this special issue focus on trustworthiness in multimedia communications. Recently, social multimedia content is being delivered to users with a high quality of experience (QoE) with the advance of multimedia technologies and social networks. However, as a huge amount of social users have various demands to exchange and share multimedia content with each other, it becomes a new challenge for the current social multimedia analytics and delivery to deal with the various attacks perpetrated by malicious users or through spam contents. Therefore, the trust and risk management for social multimedia content based on the social tie of users become of prime importance to face the unpredicted threats and subsequent damage. This Special Section aims to provide a premier forum for researchers working on the trust-based social multimedia analytics and delivery. It also provides the opportunity for both academic and industrial researchers to discuss recent results and provide solutions to the above-mentioned challenges.

Hybrid Deep-Learning-Based Anomaly Detection Scheme for Suspicious Flow Detection in SDN: A Social Multimedia Perspective

The continuous development and usage of multi-media-based applications and services have contributed to the exponential growth of social multimedia traffic. In this context, secure transmission of data plays a critical role in realizing all of the key requirements of social multimedia networks such as reliability, scalability, quality of information, and quality of service (QoS). Thus, a trust-based paradigm for multimedia analytics is highly desired to meet the increasing user requirements and deliver more timely and actionable insights. In this regard, software-defined networks (SDNs) play a vital role; however, several factors such as as-runtime security, and energy-aware networking limit its capabilities to facilitate efficient network control and management. Thus, with the view to enhance the reliability of the SDN, a hybrid deep-learning-based anomaly detection scheme for suspicious flow detection in the context of social multimedia is proposed. It consists of the following two modules: 1) an anomaly detection module that leverages improved restricted Boltzmann machine and gradient descent-based support vector machine to detect the abnormal activities, and 2) an end-to-end data delivery module to satisfy strict QoS requirements of the SDN, that is, high bandwidth and low latency. Finally, the proposed scheme has been experimentally evaluated on both real-time and benchmark datasets to prove its effectiveness and efficiency in terms of anomaly detection and data delivery essential for social multimedia. Further, a large-scale analysis over a Carnegie Mellon University (CMU)-based insider threat dataset has been conducted to identify its performance in terms of detecting malicious events such as-Identity theft, profile cloning, confidential data collection, etc.

Deep Learning–Based Multimedia Analytics

The multimedia community has witnessed the rise of deep learning–based techniques in analyzing multimedia content more effectively. In the past decade, the convergence of deep-learning and multimedia analytics has boosted the performance of several traditional tasks, such as classification, detection, and regression, and has also fundamentally changed the landscape of several relatively new areas, such as semantic segmentation, captioning, and content generation. This article aims to review the development path of major tasks in multimedia analytics and take a look into future directions. We start by summarizing the fundamental deep techniques related to multimedia analytics, especially in the visual domain, and then review representative high-level tasks powered by recent advances. Moreover, the performance review of popular benchmarks gives a pathway to technology advancement and helps identify both milestone works and future directions.