Mobile Multimedia Devices Research Articles

LAE-GAN: a novel cloud-based Low-light Attention Enhancement Generative Adversarial Network for unpaired text images

With the widespread adoption of mobile multimedia devices, the deployment of compute-intensive inference tasks on edge and resource-constrained devices, particularly in the context of low-light text detection, remains a formidable challenge. Existing deep learning approaches have shown limited effectiveness in restoring images for extremely dark scenes. To address these limitations, this paper presents a novel cloud-based Low-light Attention Enhancement Generative Adversarial Network for unpaired text images (LAE-GAN) for the non-paired text image enhancement task in extremely low-light conditions. In the first stage, compressed low-light images are transmitted from edge devices to a cloud server for image enhancement. The LAE-GAN, an end-to-end network comprising a Zero-DCE and AGM-net generator, is designed with a global and local discriminator structure. The initial illumination restoration of extremely low-light images is accomplished using the Zero-DCE network. To enhance text details, we propose an Enhanced Text Attention Mechanism (ETAM) that transforms text information into a comprehensive text attention mechanism across the entire network. The Sobel operator is employed to extract text edge information, while attention is focused on text region details through constraints imposed on the attention map and edge map. Additionally, an AGM-Net module is integrated to reduce noise and fine-tune illumination. In the second stage, the cloud server makes decisions based on user requirements and processes requests in parallel, scaling with the quantity of requests. In the third stage, the enhanced results are transmitted back to edge devices for text detection. Experimental results on widely used LOL and SID low-light datasets demonstrate significant improvements in both quantitative and qualitative analysis, surpassing state-of-the-art enhancement methods in terms of image restoration and text detection.

The impact of multimedia mobile terminal devices on students' comprehensive learning quality

The impact of multimedia mobile terminal devices on students' comprehensive learning quality

Hierarchical Model Compression via Shape-Edge Representation of Feature Maps—an Enlightenment From the Primate Visual System

The cumbersome computation of deep neural networks (DNNs) limits the practical deployment of DNNs on resource-constrained mobile multimedia devices. To deploy DNNs on devices with limited power and computing resources, model compression techniques are leveraged to accelerate the networks, where network pruning can improve the inference efficiency for deployed DNNs by removing redundant weights and structures. As one of the important components of DNNs, the feature maps (FMs) can be leveraged to evaluate the importance of network structures for DNN pruning. However, previous methods neglect to fully explore the characteristics of FMs to prune the DNNs. In this paper, we investigate the high capacity and resource efficient analogy-ventral dual-pathway primates visual system to propose a hierarchical pruning framework (dubbed as HPSE). In an efficient primate visual system (PVS), the analog pathway analyzes low-frequency information to facilitate the high-frequency information inference in ventral stream. In HPSE, we extract the low-frequency shape information and high-frequency edge information from FMs to present a novel pruning pipeline that resembles the analysis mechanism of PVS. In particular, we first imitate the analogy pathway to group different FMs in each layer by calculating the shape-feature overlap. Secondly, we leverage the edge information modulated by the grouping results of the first step to prune the network. The effectiveness of HPSE is verified by pruning various DNNs on different benchmarks, and the results have achieved satisfactory performances. For example, for ResNet-56 on CIFAR-10, HPSE reduces 52.9% of FLOPs with a slight accuracy improvement; for ResNet-50 on ImageNet, we achieve 54.3%-FLOPs drop with only 0.49% Top-1 accuracy loss.

Evaluating the Impact of Architectural Decisions on the Energy Efficiency of FDCT/IDCT Configurable IP Cores

The development of mobile multimedia devices follows the platform-based design methodology in which IP cores are the building blocks. In the context of mobile devices there is a concern of battery lifetime which leads to the need of energy-efficient IP cores. This paper presents four energy-efficient FDCT/IDCT configurable IP cores. These architectures are based on Massimino’s algorithm, which was chosen due to its high accuracy and parallelism. The four architectures were built by combining fully-combinational or pipelined datapaths, using either a single or two 1-D DCT blocks with a transpose buffer that assures the optimal minimum latency of eight cycles. Synthesis results for 90nm showed that our most efficient architecture, which uses two pipelined 1-D blocks, achieved 250 MHz as maximum frequency at a total power of 14.03 mW. Such frequency was enough to process 16x 1080p@30fps videos in real time (nearly 2 GigaPixels/s). Comparisons with related work, in terms of energy efficiency (μJ/MPixels), revealed that our most energy-efficient architecture is at least 2 times as efficient as other DCT architectures. Moreover, the four designed architectures were also synthesized by using common low-power techniques. These results showed that pipelined versions at high throughput tend to take more benefit from using Low-Vdd and High-Vt combined than the combinational ones, thus becoming the most energy efficient.

Extended reality interactive wall: User experience design research-creation

This article presents the development of an extended reality (XR) interactive wall design that employs augmented reality (AR) and virtual reality (AR) to display digital content in educational and exhibition settings. A wall design printed with a number of conventional graphical elements was enhanced with AR and VR, including AR-based video, a 360 virtual environment and 3D-scanned materials. The early design of the XR interactive wall was tested several times in a university setting during academic events designed to demonstrate science, technology, engineering and mathematics (STEM)-based content. This provided AR and VR experiences to participating visitors in an exhibition setting, where multimedia mobile devices were provided to aid these experiences. Inclusive design principles were employed, and elements integrated that allowed users (standing adults, children, wheelchair users) to view and interact with AR and VR content conveniently. After the initial introduction of the interactive wall design, the prototype was redesigned to fit both handheld and head-mount-device (HMD)-based mobile configurations. In this article we describe the prototype, a user experience study based on laboratory testing and directions for future work.

An audiovisual interface-based drumming system for multimodal human–robot interaction

This paper presents a study of an audiovisual interface-based drumming system for multimodal human–robot interaction. The interactive multimodal drumming game is used in conjunction with humanoid robots to establish an audiovisual interactive interface. This study is part of a project to design robot and avatar assistants for education and therapy, especially for children with special needs. It specifically focuses on evaluating robot/virtual avatar tutors, tangible interaction devices, and mobile multimedia devices within a simple drumming-based interactive music tutoring game scenario. Several parameters, including the effect of the embodiment of the tutor/interface and the presence of feedback and training mechanisms, were the focus of interest. For that purpose, we created an interactive drumming game relying on turn-taking and imitation principles, in which a human user is able to play drums with a humanoid robot (Robotic Drum Mate). Three interactive scenarios with different experimental setups for humanoid robots and mobile devices were developed and tested. As a part of those scenarios, a system that enables drum strokes to be automatically detected and recognized using auditory cues was implemented and incorporated into the experimental framework. We verified the applicability and effectiveness of the proposed system in a drum-playing game with adult human test subjects by evaluating it both objectively and subjectively. The results showed that the physical robot tutor, the feedback and training mechanisms had a positive effect on the subjects’ performance and, as expected, although the physical medium is preferred, the virtual medium for drumming caused less failure.

Security of Mobile Multimedia Data:The Adversarial Examples for Spatio-temporal Data

With the rapid development of the internet, the number of mobile multimedia devices is increasing among people. As the popularity of these mobile devices, a large amount of multimedia data is produced. How to utilize these multimedia data is essential in the wireless communication area. In recent years, deep learning has achieved good performance in Multimedia data analysis. However, recent research shows that deep learning models are vulnerable to the crafted samples. The crafted samples are called adversarial examples. In the paper, we explore the security of deep learning in mobile multimedia computing systems. We use the Spatio-temporal data (GPS trajectory data) to conduct experiments. The deep learning networks attacked are trajectory mode classification models. Since both Spatio-temporal data and image data are continuous, we adopt the algorithms of Computer Vision (Fast Gradient Sign Method (FGSM)) to generate adversarial examples. We propose an Adversarial Sample generation model based on a Convolutional AutoEncoder (ASCAE). Firstly, we design a convolutional AutoEncoder (CAE) to convert trajectory data into image format data. Then, we utilize image format data to train trajectory mode classification models. Based on image format data and the trained models, we use FGSM algorithm to generate adversarial examples. Finally, we utilize adversarial examples to attack the trained models. Experiment results demonstrate that the adversarial examples successfully attack the deep learning models. Besides, the adversarial examples have good transferability. Therefore, in mobile multimedia computing systems, adversarial examples are a serious data security problem for deep learning models.

An Efficient Implementation of Parallel Parametric HRTF Models for Binaural Sound Synthesis in Mobile Multimedia

The extended use of mobile multimedia devices in applications like gaming, 3D video and audio reproduction, immersive teleconferencing, or virtual and augmented reality, is demanding efficient algorithms and methodologies. All these applications require real-time spatial audio engines with the capability of dealing with intensive signal processing operations while facing a number of constraints related to computational cost, latency and energy consumption. Most mobile multimedia devices include a Graphics Processing Unit (GPU) that is primarily used to accelerate video processing tasks, providing high computational capabilities due to its inherent parallel architecture. This paper describes a scalable parallel implementation of a real-time binaural audio engine for GPU-equipped mobile devices. The engine is based on a set of head-related transfer functions (HRTFs) modelled with a parametric parallel structure, allowing efficient synthesis and interpolation while reducing the size required for HRTF data storage. Several strategies to optimize the GPU implementation are evaluated over a well-known kind of processor present in a wide range of mobile devices. In this context, we analyze both the energy consumption and real-time capabilities of the system by exploring different GPU and CPU configuration alternatives. Moreover, the implementation has been conducted using the OpenCL framework, guarantying the portability of the code.

SPIHT Algorithm With Adaptive Selection of Compression Ratio Depending on DWT Coefficients

In mobile multimedia devices, the frame memory compression (FMC) technique by embedded compression (EC) is becoming an increasingly important video-processing method for reducing the external data bandwidth requirement, which, in turn, results in power savings. Among various EC schemes, the combination of discrete wavelet transform (DWT) and set partitioning in hierarchical trees (SPIHT) is widely used for FMC because it achieves high compression efficiency with low computational complexity. However, there is room for improvement in the conventional DWT and SPIHT algorithm because it compresses all blocks with the same compression ratio without taking into account the correlation between DWT coefficients and the SPIHT algorithm. This study proposes a novel one-dimensional (1-D) DWT and SPIHT algorithm, which enhances the quality of the compressed video by internally applying an adaptive compression ratio for the SPIHT algorithm based on DWT coefficients while keeping the same bit-stream size. The block complexity is predicted from the distribution of DWT coefficients. Then, simple blocks are aggressively compressed with a low compression ratio, while the complex blocks are compressed with a high ratio. Furthermore, to achieve the best video quality, each compression ratio is decided by an optimization technique based on mathematical formulation. Precisely, the logarithm of mean squared error by the SPIHT algorithm is assumed to be linearly correlated with the logarithm of processed DWT coefficients. Experimental results are provided that support the aforementioned model. Compared to the conventional 1-D DWT and SPIHT algorithm, the proposed scheme remarkably improves the video quality by an average of 2.23 dB in peak signal-to-noise ratio when the target compression ratio for the SPIHT algorithm is 5/16.

Special Issue on Mobile Multimedia Big Data Embedded Systems

We are now living in an era of multimedia data. The past decades have witnessed an unprecedented upsurge in “big data” in China. Mobile multimedia devices are becoming mainstream, promoting the conversion of big data technologies into practices. The deployment of big data provides people with great opportunities to deal with what were formerly great challenges. Against this background, research is ongoing to explore further possibilities in mobile multimedia developing. Mobile Multimedia Big Data Embedded Systems are transforming science, engineering, healthcare, medicine, finance, business, and ultimately our society itself. This special issue focuses on the state-of-the-art initiatives and the great promise in this field. We thank all of the authors who submitted papers for this special section. We are also highly appreciative of the reviewers who provided valuable review feedback on the submissions. The editor hopes that this special issue will attract researchers’ interest and contribute to further developments in this field.

Mobile Apps in Daily Learning Activities

<span>The use of mobile devices is increasing in daily learning activities, providing single-person use combined with interactive learning materials, simulations, voice recognition and educational games. Mobile app stores bring to learners a large amount of mobile applications, but their value and effectiveness to support these learning activities is far from being adequately studied.</span><br /><span>In this paper, we present our recent study related to the opportunities for smart devices to be used in school education, outlining the main obstacles and challenges. The subject of the research is students' interest in the use of multimedia mobile devices for educational purposes. The aim of the study is to establish the main determinants for the complete and purposeful application of smart devices in school education.</span>

Optimal Selection of SRAM Bit-Cell Size for Power Reduction in Video Compression

In mobile multimedia devices with video compression capability, a large amount of power consumption is incurred by storing video data in static random-access memories (SRAMs). The power consumption by SRAM access is reduced by decreasing the supply voltage but the decreased supply voltage may cause data loss stored in the SRAM. To reduce the probability of data loss, previous research attempts to increase the SRAM cell size, which incurs a large area overhead. To minimize the area penalty by increasing the SRAM cell size, this paper proposes a novel method to adaptively select the cell sizes of SRAMs according to their sensitivities to the quality degradation in video compression while maintaining the total SRAM area. In order to decide the optimal size for embedded SRAMs, optimization based on Lagrange multipliers and Karush–Kuhn–Tucker conditions is proposed. The proposed optimization significantly reduces the number of simulations necessary to determine the optimal combination of SRAM bit-cell sizes. By applying the proposed heterogeneous SRAM cell sizing scheme along with the proposed optimization scheme, the best Bjontegaard delta peak signal-to-noise ratio (BDPSNR) improvement is achieved. Simulation results show that the proposed approach remarkably improves the video quality by up to 3.72 dB in BDPSNR compared with the conventional SRAM with an identical cell size. These results imply that the proposed heterogeneous SRAM allows a reduction of the supply voltage while maintaining the video quality.

Slack-Time Closed-Loop Control System for Multimedia Mobile Devices

Nowadays there is a pervasive utilization of consumer-electronics mobile devices with powerful multimedia capabilities. The suitable management of their power consumption is critical in order to meet the user expectations in terms of battery lifetime. But the reduction of the dynamic-power component of the consumption may sometimes impair the quality of multimedia experience (QoE), which is not always acceptable. This paper proposes a closed-loop control system that can monitor the QoE in the form of the slack time of video decoding tasks and act on the CPU frequency to regulate it. Some results are presented in comparison with dynamic Linux governors which show a suitable behavior in guaranteeing enough QoE, even with power savings of almost 5% and low overhead. This enables its use as an assistant part of dynamic-power savers.

Pre-Service English Teachers’ Mobile Information and Communication Technology-Technological Pedagogy and Content Knowledge

This study investigated how pre-service English teachers develop and use MICT (mobile information and communication technology)-TPACK (technological pedagogy and content knowledge) to select and use mobile applications for their micro teaching in a pre-service teacher training program. Although mobile technology has rapidly developed with the adoption of mobile multimedia devices and applications, the use of pre-service teachers’ TPACK is still limited to lower-level searches and to a mere tool for content presentations. The participants were nine students in a pre-service teacher training course in a four-year women’s university. Several research methods such as surveys, participant observations, micro teaching and interviews were utilized. The results of the study show the concept of TPACK needs to be extended to MICT-TPACK in this mobile age. The use of pre-service English teachers’ MICT-TPACK impacted and changed their concept of pedagogy to heutagogy. The teachers used mobile applications to facilitate their students’ inquiry-based learning of English as a subject as well as a medium of digital literacy. In order to select mobile applications for their lesson, the teachers developed a modified version of quality criteria for mobile applications. This study suggests there should be well-developed quality criteria for evaluating affordances of mobile applications.

Closed-loop power-control governor for multimedia mobile devices

The micro-electronics industry has been boosting the capabilities of multimedia mobile devices, but the battery, which is the only power source of most mobile devices, is experiencing relatively slow development. Therefore, determining how to optimize the energy consumption of mobile devices under a predefined performance requirement has become a critical issue. The video decoder, as one of the main energyconsuming multimedia applications, is the target application of this manuscript. This paper presents control algorithms for power regulation under the limited battery capacities of multimedia hand-held devices while executing a decoder application and maintaining a reasonable quality of user experience. A control system, which includes a real-time closedloop control subsystem and a power-control governor (PCG), has been implemented in the operating system of a low-cost development board. Instead of using any specific power sensor, a PMC (performance monitoring counter)-based estimator is used as the feedback signal in the closed-loop subsystem. After a theoretical system model has been obtained, classic controllers have been implemented in the development board. The control system is able to regulate the power consumption and the battery discharge rate in the presence of fluctuations in the decoder power-consumption demand. The proposed PCG can extend the battery lifetime by 15.5% and 12.8% in comparison with the conservative and ondemand governors of Linux, respectively.

IEEE Access Special Section Editorial: Recent Advances in Socially-Aware Mobile Networking

Mobile data traffic has been growing exponentially over the past few years. A report from Cisco shows that the mobile data traffic in 2014 grew 69 percent and was nearly 30 times the size of the entire global Internet in 2000 [item 1) in the Appendix]. One of the primary contributors to the explosive mobile traffic growth is the rapid proliferation of mobile social applications running on multimedia mobile devices (particularly smartphones). These sharp increases in mobile traffic (particularly from mobile social applications) are projected to continue in the foreseeable future. As mobile networks by and large are designed and deployed to meet people’s social needs, people’s behaviors and interactions in the social domain will shape their ways to access mobile services. Therefore, there is an urgent need to integrate social effects into the design of mobile networks.

Real-time power-consumption control system for multimedia mobile devices

Energy consumption in battery operated mobile devices is a critical issue. Autonomy periods are increasingly compromised as the complexity of tasks, such as the widespread video decoding ones, is also increased. While new higher-capacity batteries are being developed, techniques to control, i.e. save, the energy consumption can provide a significant benefit. This paper presents the proposal, implementation and test of a real-time closed-loop control system applicable to the power-consumption regulation of multimedia mobile devices. The novel closed loop does not need any specific power sensor but only standard performance monitoring counters (PMCs) available in common mobile processors. From them, the power consumption is estimated with an algorithm to be included into the operating system, along with the control system itself. A theoretical system model has been first developed to (1) enable the design of some controllers and (2) validate the actual results obtained from the control system prototype into a given low-cost development board. These results prove the control-system stability and the comparative suitability of classic controllers to keep the multimedia-system consumption close to the desired value in steady state with settling times as short as a single sample interval.

Operating System of Multimedia Devices in the Educational Process at Moscow State University of Printing Arts of Ivan Fedorov

Objectives: The relevance of the study is due to the fact that in the course of training at Moscow State University of Printing Arts of Ivan Fedorov in the field of information technology, until recently, traditional PCs with classical operating systems (Windows XP, Windows 7, etc.) have been used. Method: The main methods to investigate and solve this problem are modern information technologies, the introduction of which in publishing industry and journalism calls for implementation of comprehensive training in the learning process, including in the area of system software – the open mobile operating systems. Findings: This article is aimed at the revealing of the latest advances in the field of information technologies which allowed bringing a variety of multimedia devices (tablets, smart phones, etc.) to the market of technological tools, which have gained wide popularity among young people around the world. Mobile multimedia devices are supplied by manufacturers with pre-installed operating systems, which enable users to solve a variety of problems. One of these operating systems for mobile devices is Ubuntu, which in our view is convenient system software for educational institutions worldwide. Ubuntu operating system belongs to the class of Linux-like systems that do not require the acquisition of a license for use in the educational process. Improvements: The article reveals the peculiarities of this operating system and gives an example of applying such system software for mobile multimedia devices that appear in the publishing systems and applications.

A novel coupled meandered internal monopole antenna for UHF band

In this article, a novel coupled miniaturized low-frequency wideband internal monopole antenna for UHF band is proposed. The antenna is compact, and the coupling structure with multilayer and multiline can optimize the gain bandwidth. The prototype of the proposed antenna was fabricated, tested and discussed. Simulation and measurement results reveal that the proposed antenna can provide low return loss and high gain in the UHF band (474–742 MHz). The proposed antenna with a small size of 50 × 12 × 7 mm3 is suitable for today’s mobile multimedia broadcasting phones and other hand-held devices.

“After 300 meters turn right” – the future of Russia’s GLONASS and the development of global satellite navigation systems

Purpose – Global navigation satellite systems (GNSS) were designed to determine the exact location of objects on land, water and air for military purposes. With the opening of the satellite signal for civilian use, the technology created business opportunities for various applications. Today, satellite positioning technology is used by transporters, carriers, motorists, surveyors, builders, foresters, etc. through a wide array of devices like mobile phones or multimedia devices with built-in receiver modules. Design/methodology/approach – This paper provides the results of a recently held foresight exercise on the future development of Russia’s GLONASS system. Findings – The foresight exercise suggested a number niche markets where the GLONASS technology could be of great use, like monitoring of buildings and construction sides or the monitoring of shipments. In addition, in the case of Russia, large-scale government-driven investment programs will be key drivers for GLONASS’ growth perspectives. Originality/value – The paper provides a comprehensive picture of the development of GNSS for civilian use until 2020.