Resolution Frame Research Articles

Deep Reinforcement Learning-Based Video Offloading and Resource Allocation in NOMA-Enabled Networks

With the proliferation of video surveillance system deployment and related applications, real-time video analysis is very critical to achieving intelligent monitoring, autonomous driving, etc. Analyzing video stream with high accuracy and low latency through the traditional cloud computing represents a non-trivial problem. In this paper, we propose a non-orthogonal multiple access (NOMA)-based edge real-time video analysis framework with one edge server (ES) and multiple user equipments (UEs). A cost minimization problem composed of delay, energy and accuracy is formulated to improve the quality of experience (QoE) of the UEs. In order to efficiently solve this problem, we propose the joint video frame resolution scaling, task offloading, and resource allocation algorithm based on the Deep Q-Learning Network (JVFRS-TO-RA-DQN), which effectively overcomes the sparsity of the single-layer reward function and accelerates the training convergence speed. JVFRS-TO-RA-DQN consists of two DQN networks to reduce the curse of dimensionality, which, respectively, select the offloading and resource allocation action, as well as the resolution scaling action. The experimental results show that JVFRS-TO-RA-DQN can effectively reduce the cost of edge computing and has better performance in terms of convergence compared to other baseline schemes.

Dual feature enhanced video super-resolution network based on low-light scenarios

Video super-resolution (VSR) reconstruction technique aims to improve the spatiotemporal resolution of consecutive low resolution (LR) frames. Most of previous methods use the correlation of inter-frame to restore pixels. It is still a challenge to exploit intra-frame and inter-frame correlations to recover high-resolution (HR) frames, especially reconstruction of video in low-light conditions for sharp imaging results has been a bottleneck in current industrial environments. Therefore, this paper proposes a novel VSR network. Specifically, we first generate the hidden information of the current frame by using the appropriate combination of the front and rear frames. Then, a re-fusion block (RFB) is designed, which utilizes the hidden information to re-fuse with corresponding LR frame. After that, we integrate an improved dual attention mechanism (DAM) into network to extract more accurate feature of intra-frame without increasing the number of parameters. The technology has important practical significance in low-light and dim scenes, so we collect some industrial video sequences and make datasets to complete VSR task. Experimental results show that our model significantly outperforms state-of-the-art methods in performance.

A Cloud Computing Based Deep Compression Framework for UHD Video Delivery

Ultra-high-definition (UHD) videos has increased popularity. However, the data size of UHD videos is 4-16 times larger of HD videos. This brings many challenges to existing video delivery systems, such as the shortage of network bandwidth resources and longer network transmission latency. In this paper, we propose a cloud computing based deep compression framework named Pearl, which utilizes the power of deep learning and cloud computing to compress UHD videos. Pearl compresses UHD videos from two respects: the frame resolution and the colorful information. In pearl, an optimal compact representation of the original UHD video is learned with two deep convolutional neural networks (DCNNs): super resolution CNN (SR-CNN) and colorization CNN (CL-CNN). SR-CNN is used to reconstruct a high-resolution video from a low-resolution video, while CL-CNN is adopted to preserve the color information of the video. Pearl focuses on video content compression in two new directions. Thus, it can be integrated with any existing video compression system. We evaluate the performance of Pearl with a wide variety of network conditions, quality of experience metrics, and video properties. In all considered scenarios, Pearl can further compress 84% of video size and reduce 73% of network transmission latency.

Benchmarking of Contactless Heart Rate Measurement Systems in ARM-Based Embedded Platforms

Heart rate monitoring is especially important for aging individuals because it is associated with longevity and cardiovascular risk. Typically, this vital parameter can be measured using wearable sensors, which are widely available commercially. However, wearable sensors have some disadvantages in terms of acceptability, especially when used by elderly people. Thus, contactless solutions have increasingly attracted the scientific community in recent years. Camera-based photoplethysmography (also known as remote photoplethysmography) is an emerging method of contactless heart rate monitoring that uses a camera and a processing unit on the hardware side, and appropriate image processing methodologies on the software side. This paper describes the design and implementation of a novel pipeline for heart rate estimation using a commercial and low-cost camera as the input device. The pipeline's performance was tested and compared on a desktop PC, a laptop, and three different ARM-based embedded platforms (Raspberry Pi 4, Odroid N2+, and Jetson Nano). The results showed that the designed and implemented pipeline achieved an average accuracy of about 96.7% for heart rate estimation, with very low variance (between 1.5% and 2.5%) across processing platforms, user distances from the camera, and frame resolutions. Furthermore, benchmark analysis showed that the Odroid N2+ platform was the most convenient in terms of CPU load, RAM usage, and average execution time of the algorithmic pipeline.

Learning Spatiotemporal-Selected Representations in Videos for Action Recognition

Action recognition is a challenging task of modeling both spatial and temporal context. Numerous works focus on architectures modality and successfully make worthy progress on this task. While due to the redundancy in time and the limit of computation resources, several works focus on the efficiency study like frame sampling, some for untrimmed videos, and some for trimmed videos. With the intent of improving the effectiveness of action recognition, we propose a novel Computational Spatiotemporal Selector (CSS) to refine and reinforce the key frames with discriminative information in video. Specifically, CSS includes two modules: Temporal Adaptive Sampling (TAS) module and Spatial Frame Resolution (SFR) module. The former can refine the key frames in the temporal space for capturing the key motion information, while the latter can further zoom out some refined frames in the spatial space for eliminating the discrimination-irrelevant structural information. The proposed CSS is flexible to be embedded into most representative action recognition models. Experiments on two challenging action recognition benchmarks, i.e., ActivityNet1.3 and UCF101, show that the proposed CSS improves the performance over most existing models, not only on trimmed videos but also untrimmed videos.

Identifications of the coefficients of the Taylor expansion (second order) of periodic non-collision solutions for the perturbed planar Keplerian Hamiltonian system

<abstract><p>The discussion of disordered Keplerian Hamiltonian systems in our previously published study, which we verified, is expanded upon in this article. The collision semicircular orbit, and at least one other symmetric orbit is mentioned in this article. The proofs are based on the circular orbital decomposition and implicit function theory, and they concur with the results provided by Ambrosetti A. and Coyi Zelati. In the second stage, I use the Lindsted-Poincar approach to discover an asymptote. The Taylor squared expansion coefficients for periodic solutions of non-collision, are now defined.This interferes With the Kepler-Hamiltonian system, which is a part of the planar Kepler-Hamiltonian system. Systems with perturbations execute a Taylor expansion of the modulus when a system is perturbed in the time frame of full resolution and the word epsilon.</p></abstract>

Energy-Efficient Task Offloading of Edge-Aided Maritime UAV Systems

This paper considers the autonomous detecting and tracking task of unmanned aerial vehicles (UAVs) in the maritime environment. Due to the high computational complexity of the image-processing task, the strict latency requirement of the tracking task, and the shortage of batteries and computational capability of UAVs, the UAVs can offload the computing-intensive task to the edge computing server (ECS) to reduce the task latency and energy consumption. Even though, the large latency and energy consumption brought by some high-definition image-processing task still make some visual tracking difficult to perform. We first investigate the impact of video frame resolution on computing task size and detection accuracy, and further on task latency and energy consumption. In addition, we optimize the transmission power, the local CPU frequency, the offloading rate of UAVs, and the bandwidth allocation between UAVs. We model the UAVs tracking system as an energy consumption optimization problem with the constraint of task latency. Although the proposed optimization problem is complex, we transform the optimization problem into several subproblems with decoupling and rigorous mathematical proof. By combining the convex optimization with the genetic algorithm, an offloading algorithm with low complexity and good performance is obtained, which is verified by comparing with some benchmark offloading strategies in the numerical results.

Scale Invariant low frame rate tracking

The cameras used for road surveillance present low image quality and significant variability of objects’ scale, making real-time vehicle detection and tracking challenging. This paper proposes Scale Invariant Low Frame Rate Tracking, called SILFT, a new methodology to track vehicles in most surveillance systems whose frame rate and resolution are low. This architecture uses the fusion of a dense optical flow with object detection in a 6-step pipeline. The detections are masks for the optical flow that later will be used for finding the object correspondence between frames, making it effective in low frame rate sequences with large object scale variability. SILFT achieved average precision for detection of 65.97%, surpassing the YOLO and FASTER R-CNN models in the proposed database. For the tracking task, SILFT achieved a PR-MOTA of 0.45, whereas the YOLO model with intersection over union tracking achieved 0.13 in the proposed dataset.

HEVC Based Frame Interleaved Coding Technique for Stereo and Multi-View Videos

The standard HEVC codec and its extension for coding multiview videos, known as MV-HEVC, have proven to deliver improved visual quality compared to its predecessor, H.264/MPEG-4 AVC’s multiview extension, H.264-MVC, for the same frame resolution with up to 50% bitrate savings. MV-HEVC’s framework is similar to that of H.264-MVC, which uses a multi-layer coding approach. Hence, MV-HEVC would require all frames from other reference layers decoded prior to decoding a new layer. Thus, the multi-layer coding architecture would be a bottleneck when it comes to quicker frame streaming across different views. In this paper, an HEVC-based Frame Interleaved Stereo/Multiview Video Codec (HEVC-FISMVC) that uses a single layer encoding approach to encode stereo and multiview video sequences is presented. The frames of stereo or multiview video sequences are interleaved in such a way that encoding the resulting monoscopic video stream would maximize the exploitation of temporal, inter-view, and cross-view correlations and thus improving the overall coding efficiency. The coding performance of the proposed HEVC-FISMVC codec is assessed and compared with that of the standard MV-HEVC’s performance for three standard multi-view video sequences, namely: “Poznan_Street”, “Kendo” and “Newspaper1”. Experimental results show that the proposed codec provides more substantial coding gains than the anchor MV-HEVC for coding both stereo and multi-view video sequences.

Deep Learning Video Analytics Through Online Learning Based Edge Computing

Video analytics demand intensive computation resources, which means long processing delay when running on mobile devices. Although offloading computation to the cloud can partially solve the problem, transferring videos to the cloud introduces high transmission delay. With mobile edge computing, computation can be offloaded to the nearby edge servers to reduce the delay. However, the computation resources of the edge servers are usually limited and highly dynamic, and then server selection should be adaptive in order to improve the performance of video analytics. Also, frame resolution should be selected to achieve a better tradeoff between accuracy and frame processing rate. In this paper, we study the server resource-aware offloading problem for video analytics, where the goal is to maximize the utility which is a weighted function of accuracy and frame processing rate. The major challenge to solve this problem is the lack of server and network knowledge and the dynamic system environment. To overcome these challenges, we formulate the problem as a contextual Multi-armed Bandit problem, and propose a Bayesian Optimization based online learning algorithm to gradually learn the server status and the optimal solution, and make it adaptable for time-varying environments. Both theoretical analysis and evaluation results demonstrate the superior performance of our proposed algorithm.

SAMU-XLSR: Semantically-Aligned Multimodal Utterance-Level Cross-Lingual Speech Representation

We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework. Unlike previous works on speech representation learning, which learns multilingual contextual speech embedding at the resolution of an acoustic frame (10-20ms), this work focuses on learning multimodal (speech-text) multilingual speech embedding at the resolution of a sentence (5-10s) such that the embedding vector space is semantically aligned across different languages. We combine state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R with the Language Agnostic BERT Sentence Embedding (LaBSE) model to create an utterance-level multimodal multilingual speech encoder SAMU-XLSR. Although we train SAMU-XLSR with only multilingual transcribed speech data, cross-lingual speech-text and speech-speech associations emerge in its learned representation space. To substantiate our claims, we use SAMU-XLSR speech encoder in combination with a pre-trained LaBSE text sentence encoder for cross-lingual speech-to-text translation retrieval, and SAMU-XLSR alone for cross-lingual speech-to-speech translation retrieval. We highlight these applications by performing several cross-lingual text and speech translation retrieval tasks across several datasets.

DRL-Based Secure Video Offloading in MEC-Enabled IoT Networks

Wireless offloading in mobile-edge-computing (MEC)-enabled Internet of Things (IoT) networks inevitably suffers the risk of eavesdropping. Physical-layer security (PLS) approaches can be applied to prevent eavesdropping. However, the existing PLS techniques are not well targeted for videos due to the fact that video’s distortion characteristics, which allow encoding parameters to be flexibly adjusted to enhance security in offloading, are ignored. A deep reinforcement learning (DRL)-based real-time, secure, and efficient video offloading scheme is proposed in this article, where video frame resolution, one key parameter of video’s distortion characteristics, is introduced and jointly optimized with PLS scheme to guarantee video’s security, improve users’ Quality of Experience (QoE) and save energy consumption. We formulate a joint optimization problem of video frame resolution selection, computation offloading, and resource allocation strategy, to minimize energy consumption and maximize QoE in terms of delay and analytic accuracy, while subject to security rate, computing capability, and transmission power. To solve the formulated NP-hard problem with the form of high-dimensional nonlinear mixed-integer programming, the hierarchical reward-function-based DRL (JVFRS-CO-RA-MADDPG) algorithm is proposed to guide the agents to obtain the optimal policy efficiently. Finally, the simulation results show that the proposed algorithm outperforms the existing algorithms in terms of delay, energy consumption, and security level.

Image Stitching Technology in 3D Film Production Based on Digital Image Processing

Image stitching in 3D film production synchronizes multiple independent frame images as a single high-resolution output. The stitching process is common for displaying a large coverage or shot in a single frame containing distinct images. For improving the realistic accuracy of 3D film images, this manuscript introduces an Itinerant Pixel-Matching-based Stitching Process (IPMSP). The proposed stitching process relies on cross-sectional pixels identified in merging two or more images. Based on the linear cumulative distribution in augmenting images, the homogeneity feature is identified. If the homogeneity is high then the stitching for linear pixels occurs increasing the frame resolution and size. The cumulative distribution is determined using the recurrent neural network in verifying homogeneity and contrast. For the contrast pixels identified, the cross-sectional matching is performed for substituting similar pixels in the missing sections. The process is repeated until the stitching region is the same based on the homogeneity feature and increased dimensions. Therefore, this process is capable of improving accuracy, precision, substitution, and reducing errors and complexity.

Gated fusion network for SAO filter and inter frame prediction in Versatile Video Coding

In order to achieve higher coding efficiency, the Versatile Video Coding (VVC) standard includes several new components at the expense of an increase in decoder computational complexity. These technologies often create ringing and contouring effects on the reconstructed frames at a low bit rate and introduce blurring and distortion. To smooth those visual artifacts, the H.266/VVC framework supports four post-processing filter operations. The state-of-the-art CNN-based in-loop filters prefer to deploy multiple networks for various quantization parameters and frame resolutions, which increases training resources and subsequently becomes overhead at decoder frame reconstruction. This paper presents a single deep-learning-based model for sample adaptive off-set (SAO) non-linear filtering operation on the decoder side, uses feature correlation among adjacent frames, and substantiates the merits of intra–inter frame quality enhancement. We introduced a variable filter size dual multi-scale convolutional neural network (D-MSCNN) to attenuate the compression artifact and incorporated strided deconvolution to restore the high-frequency details on the distorted frame. Our model follows sequential training across all QP values and updates the model weights. Using data augmentation, weight fusion, and residual learning, we demonstrated that our model could be trained effectively by transferring the convolution prior feature indices to the decoder to produce a dense output map. The Objective measurements demonstrate that the proposed method outperforms the baseline VVC method in PSNR, MS-SSIM, and VMAF metrics and achieves an average of 5.16% bit rate saving on different test sequence categories.

Visible imaging system with changeable field of view on the HL-2A tokamak.

A new visible imaging system characterizing a flexible optical design and delivering high resolution frames is established on the HL-2A tokamak. It features a modular configuration, consisting of a front-end imaging lens, a set of bilateral telecentric relay lenses, and a camera. To avoid the effects of plasma radiation (x and gamma-rays) and magnetic field variation on the camera, it should be away from the coils. Therefore, the length of the relay lenses determines the total size of the imaging system. The main feature of this imaging system is to realize the variation of field of view (FOV) by interchanging the front-end prime lenses or by using a zoom lens directly rather than designing the optical system afresh, which lowers the cost drastically. The primary purpose of varying FOV is to enrich the versatility of this system, i.e., focusing on a narrow FOV such as gas puff imaging or a wide FOV such as the plasma cross sections. During the HL-2A experiments, this visible imaging system is used to provide high quality pictures of the plasma-wall interaction, divertor detachment, pellet injections, and so on. The frames confirmed that a strong radiation close to the X point is correlated with the completely detached inner target.

Design of a plasmonic refractive index sensor based on the amplifier system with two plasmonic waveguides and four cavities with different dimensions and coordinates

In this research, we design a plasmonic refractive index sensor based on metal insulated metal waveguide (MIM) and examine it numerically. To design the structure of this sensor, we use five cavities with different dimensions and coordinates and two plasmonic waveguides. The resonant wavelengths and refractive index of the resonators are investigated and simulated by the finite difference time domain (FDTD) method. Due to the fact that the sensors and conduction characteristics of plasmonic waves are affected by the structure parameters, by changing the refractive index and changing the dimensions of the cavities, we can weaken or strengthen the passage coefficient in the resonant modes. As a result, we obtain the sensitivity coefficient, the competency digit (FOM) and the sensor quality coefficient. These plasmonic sensors with a simple frame and high optical resolution can be used to measure the refractive index in the medical, chemical and food industries, and due to the special correct distribution of resonances, they are highly sensitive to changes in the refractive index of the environment.

An Autonomous UAV-Assisted Distance-Aware Crowd Sensing Platform Using Deep ShuffleNet Transfer Learning

Autonomous unmanned aerial vehicles (UAVs) are essential for detecting and tracking specific events, such as automatic navigation. The intelligent monitoring of people’s social distances in crowds is one of the most significant events caused by the coronavirus. The virus is spreading more quickly among the crowds, and the disease cycle continues in congested areas. Due to the error that occurs when humans monitor their activity, an automated model is required to alert to social distance violations in crowds. As a result, this article proposes a two-step framework based on autonomous UAV videos, including human tracking and deep learning-based recognition of the crowd’s social distance. The deep architecture is a modified-fast and lightweight ShuffleNet learning structure. First, the Kalman filter is used to determine the positions of individuals, and then the modified ShuffleNet is used to refine the bounding boxes obtained and determine the social distance. The social distance is calculated using the initial refinement of the bounding box obtained during the tracking step and the scale in frames of the human body. The observed average accuracy, average processing time (APT), and processed frame per second (FPS) for three congestion datasets were 97.5%, 84 milliseconds, and 11.5 FPS, respectively. Real-time decision-making was achieved by reducing the size and resolution of the frames. Additionally, the frames were re-labeled to reduce the computational complexity associated with detecting social distancing. The experimental results demonstrated that the proposed method could operate more quickly and accurately on various resolution frames of UAV videos with difficult conditions.

A multi-stage spatio-temporal adaptive network for video super-resolution

Video super-resolution aims at restoring the spatial resolution of the reference frame based on consecutive input low-resolution (LR) frames. Existing implicit alignment-based video super-resolution methods commonly utilize convolutional LSTM (ConvLSTM) to handle sequential input frames. However, vanilla ConvLSTM processes input features and hidden states independently in operations and has limited ability to handle the inter-frame temporal redundancy in low-resolution fields. In this paper, we propose a multi-stage spatio-temporal adaptive network (MS-STAN). A spatio-temporal adaptive ConvLSTM (STAC) module is proposed to handle input features in low-resolution fields. The proposed STAC module utilizes the correlation between input features and hidden states in the ConvLSTM unit and modulates the hidden states adaptively conditioned on fused spatio-temporal features. A residual stacked bidirectional (RSB) architecture is further proposed to fully exploit the processing ability of the STAC unit. The proposed STAC and RSB architecture promote the vanilla ConvLSTM’s ability to exploit the inter-frame correlations, thus improving the reconstruction quality. Furthermore, different from existing methods that only aggregate features from the temporal branch once at a specified stage of the network, the proposed network is organized in a multi-stage manner. The corresponding temporal correlation in features at different stages can be fully exploited. Experimental results on Vimeo-90K-T and UMD10 datasets show that the proposed method has comparable performance with current video super-resolution methods. The code is available at https://github.com/yhjoker/MS-STAN.

A Snapshot Imaging Spectrometer Based on Uniformly Distributed-Slit Array (UDA).

Herein, we propose a system for a snapshot video hyperspectral imaging method based on a uniformly distributed-slit array (UDA) coding plate that not only effectively improves the scanning speed of spectrometers but also achieves a high spectral fidelity of snapshot videos. A mathematical model and optical link simulation of the new system are established. The analysis results show that the proposed method can more efficiently collect information and restore the spectral data cube, and the spectral smile of the system is less than 4.86 μm. The results of the spectral performance and external imaging tests of the system show that the system has the ability to collect spatial spectrum video information with a frame rate of 10 Hz and identify dynamic targets, laying a foundation for the design of a system with a higher frame rate and resolution.

Adaptive Configuration Selection and Bandwidth Allocation for Edge-Based Video Analytics

Major cities worldwide have millions of cameras deployed for surveillance, business intelligence, traffic control, crime prevention, etc. Real-time analytics on video data demands intensive computation resources and high energy consumption. Traditional cloud-based video analytics relies on large centralized clusters to ingest video streams. With edge computing, we can offload compute-intensive analysis tasks to nearby servers, thus mitigating long latency incurred by data transmission via wide area networks. When offloading video frames from the front-end device to an edge server, the application configuration (i.e., frame sampling rate and frame resolution) will impact several metrics, such as energy consumption, analytics accuracy and user-perceived latency. In this paper, we study the configuration selection and bandwidth allocation for multiple video streams, which are connected to the same edge node sharing an upload link. We propose an efficient online algorithm, called JCAB, which jointly optimizes configuration adaption and bandwidth allocation to address a number of key challenges in edge-based video analytics systems, including edge capacity limitation, unknown network variation, intrusive dynamics of video contents. Our algorithm is developed based on Lyapunov optimization and Markov approximation, works online without requiring future information, and achieves a provable performance bound. We also extend the proposed algorithms to the multi-edge scenario in which each user or video stream has an additional choice about which edge server to connect. Extensive evaluation results show that the proposed solutions can effectively balance the analytics accuracy and energy consumption while keeping low system latency in a variety of settings.