Multimedia Processing Research Articles

Unpuzzling spatio-vertical and multi-media patterns of aniline accelerators/antioxidants in an urban estuary

Aniline accelerators and antioxidants (AAs) are high-production-volume industrial additives that have recently attracted emerging concern given their ubiquity in environmental compartments and the associated (eco)toxic effects. Nonetheless, available information on the multi-media behavior of AAs and their transformation products (TPs) remains scarce. Therefore, we determined the residues of twenty-four AA(TP)s in paired dissolved phases (i.e., filtered water), suspended particulate matter (SPM), and sediment samples collected from the Yangtze River Estuary (YRE), a highly urbanized estuary in the East China. The median total concentrations of targeted compounds were 0.73 ng/g dw, 34.4 ng/L, and 39.6 ng/L in sediments, surface and bottom water, respectively. Diphenylamine (DPA) was the most abundant congener in SPM, while 1,3-diphenylguanidine (DPG) and dicyclohexylamine (DChA) dominated in the dissolved phases and sediments. Various anthropogenic emissions and (a)biotic degradation may collectively shape the matrix-specific accumulation patterns and spatial trends of these compounds across the YRE. However, the vertical patterns of AA(TP)s were obscure, probably due to the estuarine hydrodynamics and/or the modest sample size. The SPM fractions of AA(TP)s in water (Ф: 7.9–100%) and the sediment sorption coefficients (KOC: 0.01–6.56) both positively correlated with their hydrophobicity as indicated by the octanol-water partition coefficient (KOW). Moreover, risk quotients implied moderate to high aquatic toxicity posed by several AA(TP)s at certain YRE sites. The estimated total annual fluxes of our analytes transported via water and sediments towards the East China Sea were 5.90–365.5 tons and 4.23–1,100 kg, respectively. This work provides a systematic investigation of multi-media processes and ecological risks of AA(TP)s in a highly-urbanized estuary, contributing to holistic comprehension of these emerging contaminants in estuarine environments.

SMART: Supervised multi-class image retargeting generative model based on a long-range sampling strategy

Content-aware image retargeting (CAIR) techniques are crucial in multimedia processing for displaying images on various devices while preserving visually salient contents with desirable visual effects. There are discrete and continuous algorithms. For the former, the artefacts happen when the foreground proportion is larger than the retargeting ratio; for the latter, the salient regions are prone to be squeezed. In this paper, we reformulate the retargeting process into sampling the salient signal and reconstruction under aesthetic supervision, the supervised multi-class image retargeting reconstruction (SMART) framework. The target images can be represented into complementary parts, the masked and unmasked ones, according to the saliency influences in the encoder phrase. The long-range sampling algorithm is proposed to calculate similarities through an 8-connected planar path while considering spatial distance and feature correlation. The sampled embeddings in latent space reconstruct the retargeted images under supervised signals for aesthetic quality. The semantic loss Lsem from the pretrained CLIP model can maintain consistency for both content and semantics. The supervised loss, Lir, is introduced to ensure the retargeted qualities are close to the preferred labels. Then, we release a new retargeting dataset comprising seven image classes (animal, building, car, flower, indoor, landscape and people) with supervised labels collected from designers for further aesthetic retargeting study. The ablation studies are conducted to confirm the effectiveness of the new dataset, and comparative experiments with state-of-the-art baselines demonstrate the advantages of the proposed method.

Video Summarization using Object Detection Method

Video summarization is a fundamental challenge in the field of computer vision and multimedia processing, aimed at condensing lengthy videos into concise representations without compromising the essential content and context. This project focuses on the integration of object detection techniques into the process of video summarization, harnessing the power of deep learning to automatically identify and extract key objects and events from video sequences. By leveraging state-of-the-art object detection models and innovative summarization algorithms, this project aims to enhance the efficiency and effectiveness of video summarization, enabling users to quickly grasp the content and significance of videos without the need for exhaustive playback. The proposed approach not only streamlines video browsing and content comprehension but also holds potential applications in various domains, including surveillance, video indexing, and content recommendation systems

A Unified Framework for Jointly Compressing Visual and Semantic Data

The rapid advancement of multimedia and imaging technologies has resulted in increasingly diverse visual and semantic data. A large range of applications such as remote-assisted driving requires the amalgamated storage and transmission of various visual and semantic data. However, existing works suffer from the limitation of insufficiently exploiting the redundancy between different types of data. In this article, we propose a unified framework to jointly compress a diverse spectrum of visual and semantic data, including images, point clouds, segmentation maps, object attributes, and relations. We develop a unifying process that embeds the representations of these data into a joint embedding graph according to their categories, which enables flexible handling of joint compression tasks for various visual and semantic data. To fully leverage the redundancy between different data types, we further introduce an embedding-based adaptive joint encoding process and a Semantic Adaptation Module to efficiently encode diverse data based on the learned embeddings in the joint embedding graph. Experiments on the Cityscapes, MSCOCO, and KITTI datasets demonstrate the superiority of our framework, highlighting promising steps toward scalable multimedia processing.

Advancing defense capabilities through integration of electro-optical systems and computer vision technologies

This paper presents a comprehensive study on the integration of cutting-edge technologies to enhance defense capabilities, focusing on object detection, tracking, and distance measurement tasks. Also, the integration of IMX219-77 cameras and Nvidia Jetson Nano for object detection, tracking, and distance measurement in electro-optical systems is proposed. The decision to combine these components was based on their individual strengths: the high resolution and wide field of view of the IMX219-77 cameras, and the low-power, high-performance capabilities of the Jetson Nano. Software-wise, OpenCV and GStreamer were chosen for their widespread use and capabilities in computer vision and multimedia processing tasks, respectively. The MOSSE algorithm was selected for object tracking due to its speed, efficiency, and resilience to lighting changes. Additionally, Depth Estimation (Stereo Vision) was employed for distance measurement by analyzing images captured from a pair of cameras. The proposed algorithm involves transmitting images from the cameras to the Jetson Nano, where OpenCV and GStreamer are utilized to process the data. The MOSSE algorithm is employed for object detection and tracking, while Depth Estimation generates depth maps to measure distances between objects and the camera. The study's results demonstrate accurate distance measurements with a margin of error ranging from 0 to 2 mm, falling below the acceptable threshold mentioned in academic literature. This aligns with similar studies in the field, showcasing the efficacy of the proposed integration for object detection, tracking, and distance measurement in electro-optical systems.

TransIFC: Invariant Cues-aware Feature Concentration Learning for Efficient Fine-grained Bird Image Classification

Fine-grained bird image classification (FBIC) is not only meaningful for endangered bird observation and protection but also a prevalent task for image classification in multimedia processing and computer vision. However, FBIC suffers from several challenges, such as bird molting, complex background, and arbitrary bird posture. To effectively tackle these challenges, we present a novel invariant cues-aware feature concentration Transformer (TransIFC), which learns invariant and core information in bird images. To this end, two novel modules are proposed to leverage the characteristics of bird images, namely, the hierarchy stage feature aggregation (HSFA) module and the feature in feature abstraction (FFA) module. The HSFA module aggregates the multiscale information of bird images by concatenating multilayer features. The FFA module extracts the invariant cues of birds through feature selection based on discrimination scores. Transformer is employed as the backbone to reveal the long-dependent semantic relationships in bird images. Moreover, abundant visualizations are provided to prove the interpretability of the HSFA and FFA modules in TransIFC. Comprehensive experiments demonstrate that TransIFC can achieve state-of-the-art performance on the CUB-200-2011 dataset (91.0%) and the NABirds dataset (90.9%). Finally, extended experiments have been conducted on the Stanford Cars dataset to suggest the potential of generalizing our method on other fine-grained visual classification tasks.

Robust Image Watermarking using DWT, DCT, and PSO with CNN-Based Attack Evaluation

Digital Content Protection is one of the most significant research areas that lies at the intersection of cyber-security and multimedia processing. Protect multimedia content from copyright violation, unauthorized use, replication, and online content theft is needed. Digital Image Watermarking is utilized to preserve the copyright of different digital images from forgery. Various techniques had been developed in this regard with two main issues, the method robustness and the resistance against various types of attacks like, Salt and Pepper noise, filtering and blurring. Current Digital watermarking techniques may reduce the quality of the original digital media content if it is not robust. The purpose of this research is to create a robust image watermarking technique against different attack types such as salt and pepper noise and Gaussian noise, ensuring the image content is protected. Specifically, this study proposed a new mechanism for Image watermarking based on combining Discrete Wavelet Transform (DWT), and Discrete Cosine Transform (DCT). Additionally, Particle Swarm Optimization (PSO) was applied to perform the optimization for both the embedding and extraction processes. At the final stage we assess the proposed approach against some types of attacks such as Additive White Gaussian Noise (AWGN). The Denoising Convolutional Neural Network, (DnCNN) was used to evaluate the mechanism against AWGN. For testing, we utilized measures such as Peak Signal-to-Noise Ratio (PSNR) and Normalized Correlation Coefficient (NCC). The experimental results of implementing our proposed method for embedding and extracting watermarks across various host image sizes were encouraging, achieving a PSNR ratio of 0.998 and an NCC of 1 in the absence of attacks. Additionally, our evaluation revealed that a specific type of denoising attack, while damaging the watermark (though not completely), actually improved the image quality. It is also important to highlight that our findings surpassed those reported in existing literature, with the PSNR and NCC values serving as evidence of this superior performance.

EHPE: Skeleton Cues-Based Gaussian Coordinate Encoding for Efficient Human Pose Estimation

Human pose estimation (HPE) has many wide applications such as multimedia processing, behavior understanding and human-computer interaction. Most previous studies have encountered many constraints, such as restricted scenarios and RGB inputs. To mitigate constraints to estimating the human poses in general scenarios, we present an efficient human pose estimation model (i.e., EHPE) with joint direction cues and Gaussian coordinate encoding. Specifically, we propose an anisotropic Gaussian coordinate coding method to describe the skeleton direction cues among adjacent keypoints. To the best of our knowledge, this is the first time that the skeleton direction cues is introduced to the heatmap encoding in HPE task. Then, a multi-loss function is proposed to constrain the output to prevent the overfitting. The Kullback-Leibler divergence is introduced to measure the predication label and its ground truth one. The performance of EHPE is evaluated on two HPE datasets: MS COCO and MPII. Experimental results demonstrate that EHPE can obtain robust results, and it significantly outperforms existing state-of-the-art HPE methods. Lastly, we extend the experiments on infrared images captured by our research group. The experiments achieved the impressive results regardless of insufficient color and texture information.

PLANTATION DISEASE DETECTION AND DIAGNOSIS

Deep learning, as a branch of AI , has got into the limelight in recent years due to its automatic learning and feature extraction capabilities. It has found widespread applications in various domains, including multimedia processing, voice analysis, and NLP. Moreover, it has emerged as a promising area of research in agricultural plant protection, specifically in tasks like plant disease recognition and weed/bacterial assessment. Leveraging neural network learning for plant disease recognition mitigates the drawbacks associated with manual process of choosing of disease marks attributes and features, processing disease feature selection and extraction more objectively and accelerating research productivity and technological advancements. This research paper encapsulates the recent advancements in DL technology concerning the identification of plantation diseases. The paper highlights current tech and their complexities in utilizing neural networks and optimized image processing techniques for plantation disease detection. We aim for this work to serve as an effective reference for researchers involved in the study of plant. Additionally, we delve into the existing challenges and issues that warrant resolution

Is animation always better for learning Chinese characters?

Abstract The present study investigated the effect of animations on Chinese character learning in multimedia presentations. In particular, we examined whether animations focusing on different knowledge of characters would yield different performance among students learning Chinese characters. The role of the animations which focused on the etymology of Chinese characters was tested in Experiment 1 with twelve English secondary school students. The results found that etymological animations were superior to flashcards for learners in recognizing forms of Chinese characters but not for understanding their meanings. In Experiment 2, the effect of animations which focused on demonstrating stroke order along with the audio explanations and on-screen written texts was addressed with 60 Thai secondary school students. The results revealed that animations demonstrating stroke order did not provide learners with advantages in orthographic learning. However, the findings support the redundant principle that static imagery and spoken and written text together facilitated character learning. In addition, the study showed that the students’ pace and flexibility of learning, the level of character difficulty, and students’ active processing have significant roles in character learning performance in a multimedia settings. This study provides implications for the design of multimedia instruction in foreign language acquisition and sheds light on the theoretical assumptions of dual-coding theory regarding learners’ cognitive loads in multimedia processing.

SMSE: A serverless platform for multimedia cloud systems

SummaryAlong with the rise of domain‐specific computing (ASICs hardware) and domain‐specific programming languages, we envision that the next step is the emergence of domain‐specific cloud platforms. Considering multimedia streaming as one of the most trendy applications in the IT industry, the goal of this study is to develop serverless multimedia streaming engine (SMSE), the first domain‐specific serverless platform for multimedia streaming. SMSE democratizes multimedia service development via enabling content providers (or even end‐users) to rapidly develop their desired functionalities on their multimedia contents. Upon developing SMSE, the next goal of this study is to deal with its efficiency challenges and develop a function container provisioning method that can efficiently utilize cloud resources and improve the users' quality of service. In particular, we develop a dynamic method that provisions durable or ephemeral containers depending on the spatiotemporal and data‐dependency characteristics of the functions. Evaluating the prototype implementation of SMSE under real‐world settings demonstrates its capability to reduce both the containerization overhead, and the makespan time of serving multimedia processing functions (by up to 30%) in compare to the function provision methods that are being used in the general‐purpose serverless cloud systems.

EMSN: An Energy-Efficient Memristive Sequencer Network for Human Emotion Classification in Mental Health Monitoring

Mental health problems are an increasingly common social issue severely affecting health and well-being. Multimedia processing technologies via facial expression show appealing prospects in the consumer field for mental health monitoring, while still suffer from intensive computation and low energy efficiency. This paper proposes an energy-efficiency memristive sequencer network (EMSN) for human emotion classification, which offers an environmentally friendly approach for consumers with low cost and easily deployable hardware. Firstly, two-dimensional (2D) materials are employed to construct an eco-friendly memristor, the efficacy and reliability of which are confirmed through performance testing. Then, a sequencer block is proposed using memristive circuits. Notably, it is a core component of the EMSN, consisting of a bidirectional long short-term memory circuit, normalisation circuit module, and multi-layer perception module. After combining some necessary function modules, the EMSN can be achieved. Furthermore, the proposed EMSN is applied for human emotion classification. The experimental results demonstrate that the proposed EMSN has advantages in computational efficiency and classification accuracy compared to existing mainstream methods, indicating an advancement in consumer health monitoring.

Efficient octagonal compression of multimedia data using using LZW-OMCA compressor for secured data transmission

Data compression is the ancestor of image compression, which uses fewer bits to represent the same picture. It is categorised as lossy or lossless depending on the quality required. In a lossless compression situation, no information is lost during the decompression process. Data loss is possible with the lossy technique since it is not reversible. In an effort to boost compression efficiency while maintaining a high xiv reconstruction quality of picture, near lossless approaches have evolved. The medical pictures consist of a large number of items, each of which may be described in detail and utilised for a variety of purposes. The clinically relevant item in 2D medical pictures is referred to as the Region of Interest (ROI), whereas in 3D images, it is referred to as the Volume of Interest (VOI). Saving energy is crucial since it is one of the most limited resources in these networks. However, DTN has an additional difficulty since communication between nodes is maintained so long as they are in physical proximity to one another. However, because to the nodes’ mobility, this may not be long enough to provide the necessary multimedia data transmission. Wireless networks are susceptible to security assaults, and traditional security solutions are computationally demanding, making them unsuitable for networks that constantly need to recharge their batteries. All of these are reasons for tackling the problems of multimedia data processing and transmission via wireless networks in this dissertation. With this in mind, it has been attempted to investigate low-overhead and safe multimedia data compression as a solution to the issue that energy-constrained nodes in these networks limit complex multimedia processing while keeping at least basic security features. LZW-OMCA compression using the Octagonal Multimedia Compression Algorithm is part of the suggested method. The purpose of this is to improve the compression ratio. The proposed approach uses a little bit of crypt to compress data, which makes the data unreadable to anybody except the intended receiver, hence providing network security. The previous proposed works analysed the performance of several compression algorithms applied to multimedia material. Performance assessment utilising MSE, SSIM, and other metrics are used to show the pros and cons of each segment.

Constraint-Aware Multi-Technique Approximate High-Level Synthesis for FPGAs

Numerous approximate computing (AC) techniques have been developed to reduce the design costs in error-resilient application domains, such as signal and multimedia processing, data mining, machine learning, and computer vision, to trade-off computation accuracy with area and power savings or performance improvements. Selecting adequate techniques for each application and optimization target is complex but crucial for high-quality results. In this context, Approximate High-Level Synthesis (AHLS) tools have been proposed to alleviate the burden of hand-crafting approximate circuits by automating the exploitation of AC techniques. However, such tools are typically tied to a specific approximation technique or a difficult-to-extend set of techniques whose exploitation is not fully automated or steered by optimization targets. Therefore, available AHLS tools overlook the benefits of expanding the design space by mixing diverse approximation techniques toward meeting specific design objectives with minimum error. In this work, we propose an AHLS design methodology for FPGAs that automatically identifies efficient combinations of multiple approximation techniques for different applications and design constraints. Compared to single-technique approaches, decreases of up to \(30\% \) in mean squared error and absolute increases of up to \(6.5\% \) in percentage accuracy were obtained for a set of image, video, signal processing and machine learning benchmarks.

Global semantic enhancement network for video captioning

Video captioning aims to briefly describe the content of a video in accurate and fluent natural language, which is a hot research topic in multimedia processing. As a bridge between video and natural language, video captioning is a challenging task that requires a deep understanding of video content and effective utilization of diverse video multimodal information. Existing video captioning methods usually ignore the relative importance between different frames when aggregating frame-level video features and neglect the global semantic correlations between videos and texts in learning visual representations, resulting in the learned representations less effective. To address these problems, we propose a novel framework, namely Global Semantic Enhancement Network (GSEN) to generate high-quality captions for videos. Specifically, a feature aggregation module based on a lightweight attention mechanism is designed to aggregate frame-level video features, which highlights features of informative frames in video representations. In addition, a global semantic enhancement module is proposed to enhance semantic correlations for video and language representations in order to generate semantically more accurate captions. Extensive qualitative and quantitative experiments on two public benchmark datasets MSVD and MSR-VTT demonstrate that the proposed GSEN can achieve superior performance than state-of-the-art methods.

From Pixels to Bits: A State of the Art of Deep Learning Approaches for Natural Image Compression

With the ever-increasing volume of digital imagery and the growing demand for efficient storage and transmission, image compression has become a crucial aspect of multimedia processing. In recent years, deep learning models have emerged as powerful tools for a wide range of computer vision tasks, including image compression. This research article presents a comprehensive state-of-the-art review of natural image compression techniques leveraging deep learning architectures.
 
 This research article presents a state-of-the-art review of natural image compression using deep learning models. With the exponential growth of digital imagery, efficient compression techniques are essential. Deep learning, particularly Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), and Autoencoders, has shown promise in this area. The paper explores various architectures, discussing their adaptation to image compression tasks. Both lossless and lossy compression approaches are surveyed, considering the trade-off between compression ratios and visual quality. The article identifies challenges, such as computational complexity, scalability, and real-world applications, while suggesting future directions.

Non-redundant shift-invariant complex wavelet transform and fractional gorilla troops optimization-based deep convolutional neural network for video watermarking

With the development of multimedia processing approaches along with the Internet leads to various issues, like data tampering, data piracy, and illicit dissemination. The quick progression of fast communication networks for digital video transmission has formulated the prerequisite copyright protection for these media. On the other hand, the development of Internet technologies has directed the accessibility of images, audio, and videos in different structures. Moreover, unlicensed users manipulate the utilization of multimedia by broadcasting them on numerous internet sites to get money fraudulently without the involvement of the original copyright holder. Besides, watermarking is a process utilized to hide the watermark sign present in the multimedia statistics that are not perceptible to an intruder for manipulating any information. This paper devised a novel video watermarking technique which uses an optimized deep learning technique. Here, the binary pixel map generation is performed using DCNN for self-embedding in input video. In addition, weights as well as bias of DCNN are trained by designed Fractional Gorilla Troops Optimization-Deep Convolutional Neural Network algorithm, thus FGTO_DCNN algorithm is developed. Accordingly, the FGTO technique is newly introduced by integrating FC along with GTO algorithm. Moreover, non-redundant Shift Invariant Complex Wavelet Transform (SICWT) is employed for extracted key frame and thus watermark image is fed into embedding process. After that, inverse non-redundant SICWT is applied for embedding video, which is further passed to key frame extraction in extraction stage. Finally, non-redundant SICWT is utilized for obtaining extracted watermark image. The developed FGTO-driven DCNN model attained improved performance than other existing video watermarking techniques with Mean Square Error (MSE) of 0.0437, Normalized Correlation Coefficient (NCC) of 0.9998, high Peak Signal-to-Noise Ratio (PSNR) value of 51.93 dB, and high Weighted Signal-to-Noise Ratio (WSNR) of 49.86 dB.

Perceptual Quality Assessment of Colored 3D Point Clouds.

3D point clouds have found a wide variety of applications in multimedia processing, remote sensing, and scientific computing. Although most point cloud processing systems are developed to improve viewer experiences, little work has been dedicated to perceptual quality assessment of 3D point clouds. In this work, we build a new 3D point cloud database, namely the Waterloo Point Cloud (WPC) database. In contrast to existing datasets consisting of small-scale and low-quality source content of constrained viewing angles, the WPC database contains 20 high quality, realistic, and omni-directional source point clouds and 740 diversely distorted point clouds. We carry out a subjective quality assessment experiment over the database in a controlled lab environment. Our statistical analysis suggests that existing objective point cloud quality assessment (PCQA) models only achieve limited success in predicting subjective quality ratings. We propose a novel objective PCQA model based on an attention mechanism and a variant of information content-weighted structural similarity, which significantly outperforms existing PCQA models. The database has been made publicly available at https://github.com/qdushl/Waterloo-Point-Cloud-Database.

LPSA: Online educational video evaluation considering cognitive and teaching style

AbstractOnline and online‐merge‐offline learning are promoting the sharing of teaching resources and the transformation of teaching and learning methods during the global epidemic. However, uneven learning resources, subject‐based classification, and incomplete result‐based evaluation always make it hard for students to keep pace with the ideal teaching objectives. Based on the theory of cognitive style and teaching style, the LPSA video classification consisting of linguistic type (L), presentative type (P), scientific type (S), and artistic type (A), is proposed to realize teaching quality evaluation and learning process detection. In addition, eight subdimensions of LPSA and its quantitative methods utilizing multimedia processing and AI algorithm are introduced to realize video classification. By automatically generating the threshold map and developing related real‐time video evaluating tools, the technical and applied feasibility of the proposed LPSA is certified, which could also facilitate student's personal learning and accurate teaching evaluation.

Context-Aware 3D Points of Interest Detection via Spatial Attention Mechanism

Detecting points of interest is a fundamental problem in 3D shape analysis and can be beneficial to various tasks in multimedia processing. Traditional learning-based detection methods usually rely on each vertex’s geometric features to discriminate points of interest from other vertices. Observing that points of interest are related to not only geometric features on themselves but also the geometric features of surrounding vertices, we propose a novel context-aware 3D points of interest detection algorithm by adopting the spatial attention mechanism in this article. By designing a context attention module, our approach presents a novel deep neural network to simultaneously pay attention to the geometric features of vertices and their local contexts during extracting points of interest. To obtain satisfactory extraction results, our method adaptively assigns different weights to those features in a data-driven way. Extensive experimental results on SHREC 2007, SHREC 2011, and SHREC 2014 datasets show that our algorithm achieves superior performance over existing methods.